2023-04-17 03:53:28,284 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba 2023-04-17 03:53:28,298 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-17 03:53:28,318 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-17 03:53:28,318 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787, deleteOnExit=true 2023-04-17 03:53:28,318 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-17 03:53:28,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/test.cache.data in system properties and HBase conf 2023-04-17 03:53:28,319 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/hadoop.tmp.dir in system properties and HBase conf 2023-04-17 03:53:28,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/hadoop.log.dir in system properties and HBase conf 2023-04-17 03:53:28,320 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-17 03:53:28,321 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-17 03:53:28,321 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-17 03:53:28,452 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-17 03:53:28,872 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-17 03:53:28,877 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-17 03:53:28,878 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-17 03:53:28,878 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-17 03:53:28,879 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 03:53:28,879 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-17 03:53:28,880 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-17 03:53:28,880 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-17 03:53:28,881 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 03:53:28,881 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-17 03:53:28,882 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/nfs.dump.dir in system properties and HBase conf 2023-04-17 03:53:28,882 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/java.io.tmpdir in system properties and HBase conf 2023-04-17 03:53:28,882 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-17 03:53:28,883 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-17 03:53:28,883 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-17 03:53:29,516 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 03:53:29,520 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 03:53:29,816 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-17 03:53:30,001 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-17 03:53:30,021 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 03:53:30,062 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 03:53:30,096 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/java.io.tmpdir/Jetty_localhost_46173_hdfs____.dr03ez/webapp 2023-04-17 03:53:30,235 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46173 2023-04-17 03:53:30,251 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-17 03:53:30,251 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-17 03:53:30,842 WARN [Listener at localhost/40881] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 03:53:30,928 WARN [Listener at localhost/40881] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-17 03:53:30,949 WARN [Listener at localhost/40881] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-17 03:53:30,957 INFO [Listener at localhost/40881] log.Slf4jLog(67): jetty-6.1.26 2023-04-17 03:53:30,963 INFO [Listener at localhost/40881] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/java.io.tmpdir/Jetty_localhost_38837_datanode____.7esz9e/webapp 2023-04-17 03:53:31,077 INFO [Listener at localhost/40881] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38837 2023-04-17 03:53:31,471 WARN [Listener at localhost/44831] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-17 03:53:32,019 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2ed4b66e5895b2c0: Processing first storage report for DS-52013a28-cded-4095-a0f0-389955fa701c from datanode 4d8c19e9-6c86-45aa-a882-f64680513502 2023-04-17 03:53:32,020 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2ed4b66e5895b2c0: from storage DS-52013a28-cded-4095-a0f0-389955fa701c node DatanodeRegistration(127.0.0.1:37359, datanodeUuid=4d8c19e9-6c86-45aa-a882-f64680513502, infoPort=42699, infoSecurePort=0, ipcPort=44831, storageInfo=lv=-57;cid=testClusterID;nsid=1288654477;c=1681703609603), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-17 03:53:32,021 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x2ed4b66e5895b2c0: Processing first storage report for DS-2cd56585-33a9-4e9b-8b79-b90c958196b8 from datanode 4d8c19e9-6c86-45aa-a882-f64680513502 2023-04-17 03:53:32,021 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x2ed4b66e5895b2c0: from storage DS-2cd56585-33a9-4e9b-8b79-b90c958196b8 node DatanodeRegistration(127.0.0.1:37359, datanodeUuid=4d8c19e9-6c86-45aa-a882-f64680513502, infoPort=42699, infoSecurePort=0, ipcPort=44831, storageInfo=lv=-57;cid=testClusterID;nsid=1288654477;c=1681703609603), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-17 03:53:32,065 DEBUG [Listener at localhost/44831] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba 2023-04-17 03:53:32,126 INFO [Listener at localhost/44831] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787/zookeeper_0, clientPort=60620, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-17 03:53:32,140 INFO [Listener at localhost/44831] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60620 2023-04-17 03:53:32,147 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:32,149 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:32,819 INFO [Listener at localhost/44831] util.FSUtils(471): Created version file at hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9 with version=8 2023-04-17 03:53:32,819 INFO [Listener at localhost/44831] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/hbase-staging 2023-04-17 03:53:33,137 INFO [Listener at localhost/44831] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-17 03:53:33,659 INFO [Listener at localhost/44831] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-17 03:53:33,696 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:33,696 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:33,697 INFO [Listener at localhost/44831] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 03:53:33,697 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:33,697 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 03:53:33,856 INFO [Listener at localhost/44831] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 03:53:33,928 DEBUG [Listener at localhost/44831] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-17 03:53:34,029 INFO [Listener at localhost/44831] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43969 2023-04-17 03:53:34,043 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:34,046 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:34,069 INFO [Listener at localhost/44831] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43969 connecting to ZooKeeper ensemble=127.0.0.1:60620 2023-04-17 03:53:34,108 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:439690x0, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 03:53:34,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43969-0x101904098950000 connected 2023-04-17 03:53:34,150 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 03:53:34,151 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 03:53:34,155 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 03:53:34,167 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43969 2023-04-17 03:53:34,167 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43969 2023-04-17 03:53:34,168 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43969 2023-04-17 03:53:34,168 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43969 2023-04-17 03:53:34,168 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43969 2023-04-17 03:53:34,177 INFO [Listener at localhost/44831] master.HMaster(444): hbase.rootdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9, hbase.cluster.distributed=false 2023-04-17 03:53:34,254 INFO [Listener at localhost/44831] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-17 03:53:34,255 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:34,255 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:34,255 INFO [Listener at localhost/44831] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-17 03:53:34,255 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-17 03:53:34,255 INFO [Listener at localhost/44831] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-17 03:53:34,260 INFO [Listener at localhost/44831] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-17 03:53:34,264 INFO [Listener at localhost/44831] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36381 2023-04-17 03:53:34,266 INFO [Listener at localhost/44831] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-17 03:53:34,272 DEBUG [Listener at localhost/44831] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-17 03:53:34,273 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:34,276 INFO [Listener at localhost/44831] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:34,278 INFO [Listener at localhost/44831] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36381 connecting to ZooKeeper ensemble=127.0.0.1:60620 2023-04-17 03:53:34,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:363810x0, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-17 03:53:34,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36381-0x101904098950001 connected 2023-04-17 03:53:34,283 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 03:53:34,284 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 03:53:34,285 DEBUG [Listener at localhost/44831] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-17 03:53:34,286 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36381 2023-04-17 03:53:34,286 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36381 2023-04-17 03:53:34,287 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36381 2023-04-17 03:53:34,287 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36381 2023-04-17 03:53:34,288 DEBUG [Listener at localhost/44831] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36381 2023-04-17 03:53:34,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:34,298 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 03:53:34,299 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:34,319 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 03:53:34,319 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-17 03:53:34,319 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:34,320 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 03:53:34,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43969,1681703612963 from backup master directory 2023-04-17 03:53:34,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-17 03:53:34,329 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:34,329 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-17 03:53:34,330 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 03:53:34,330 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:34,333 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-17 03:53:34,335 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-17 03:53:34,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/hbase.id with ID: 10a5eae4-58d9-4e0c-ad32-8798cb961bed 2023-04-17 03:53:34,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-17 03:53:34,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:34,519 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1266022f to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:34,557 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20be4b79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:34,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:53:34,590 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-17 03:53:34,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-17 03:53:34,612 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-17 03:53:34,615 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:34,619 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:34,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 03:53:34,655 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store-tmp 2023-04-17 03:53:35,084 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:35,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 03:53:35,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:53:35,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:53:35,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 03:53:35,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:53:35,085 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:53:35,085 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 03:53:35,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/WALs/jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:35,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43969%2C1681703612963, suffix=, logDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/WALs/jenkins-hbase4.apache.org,43969,1681703612963, archiveDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/oldWALs, maxLogs=10 2023-04-17 03:53:35,183 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK] 2023-04-17 03:53:35,195 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:35,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/WALs/jenkins-hbase4.apache.org,43969,1681703612963/jenkins-hbase4.apache.org%2C43969%2C1681703612963.1681703615126 2023-04-17 03:53:35,270 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK]] 2023-04-17 03:53:35,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:53:35,271 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:35,275 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,276 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,344 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,351 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-17 03:53:35,378 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-17 03:53:35,394 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:35,399 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,401 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,420 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-17 03:53:35,425 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:53:35,427 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70780860, jitterRate=0.05471700429916382}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 03:53:35,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 03:53:35,428 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-17 03:53:35,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-17 03:53:35,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-17 03:53:35,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-17 03:53:35,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-17 03:53:35,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-04-17 03:53:35,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-17 03:53:35,514 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-17 03:53:35,519 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-17 03:53:35,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-17 03:53:35,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-17 03:53:35,553 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-17 03:53:35,559 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-17 03:53:35,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-17 03:53:35,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:35,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-17 03:53:35,568 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-17 03:53:35,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-17 03:53:35,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 03:53:35,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-17 03:53:35,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:35,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43969,1681703612963, sessionid=0x101904098950000, setting cluster-up flag (Was=false) 2023-04-17 03:53:35,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:35,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-17 03:53:35,626 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:35,638 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:35,644 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-17 03:53:35,645 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:35,648 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.hbase-snapshot/.tmp 2023-04-17 03:53:35,700 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(952): ClusterId : 10a5eae4-58d9-4e0c-ad32-8798cb961bed 2023-04-17 03:53:35,707 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-17 03:53:35,715 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-17 03:53:35,716 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-17 03:53:35,720 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-17 03:53:35,721 DEBUG [RS:0;jenkins-hbase4:36381] zookeeper.ReadOnlyZKClient(139): Connect 0x2b0dd69d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:35,726 DEBUG [RS:0;jenkins-hbase4:36381] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25ca3d8c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:35,727 DEBUG [RS:0;jenkins-hbase4:36381] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21f5bded, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-17 03:53:35,756 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36381 2023-04-17 03:53:35,761 INFO [RS:0;jenkins-hbase4:36381] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-17 03:53:35,761 INFO [RS:0;jenkins-hbase4:36381] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-17 03:53:35,761 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1023): About to register with Master. 2023-04-17 03:53:35,765 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43969,1681703612963 with isa=jenkins-hbase4.apache.org/172.31.14.131:36381, startcode=1681703614253 2023-04-17 03:53:35,787 DEBUG [RS:0;jenkins-hbase4:36381] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-17 03:53:35,797 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-17 03:53:35,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:35,808 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-17 03:53:35,808 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:35,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681703645811 2023-04-17 03:53:35,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-17 03:53:35,819 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 03:53:35,820 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-17 03:53:35,828 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 03:53:35,829 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-17 03:53:35,838 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-17 03:53:35,838 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-17 03:53:35,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-17 03:53:35,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-17 03:53:35,840 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:35,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-17 03:53:35,844 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-17 03:53:35,845 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-17 03:53:35,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-17 03:53:35,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-17 03:53:35,856 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54703, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-17 03:53:35,860 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681703615850,5,FailOnTimeoutGroup] 2023-04-17 03:53:35,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681703615861,5,FailOnTimeoutGroup] 2023-04-17 03:53:35,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:35,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:35,872 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-17 03:53:35,874 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:35,875 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:35,899 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 03:53:35,900 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-17 03:53:35,900 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9 2023-04-17 03:53:35,915 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(2829): Master is not running yet 2023-04-17 03:53:35,915 WARN [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-17 03:53:35,927 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:35,937 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 03:53:35,940 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/info 2023-04-17 03:53:35,941 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 03:53:35,943 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:35,943 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 03:53:35,946 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/rep_barrier 2023-04-17 03:53:35,947 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 03:53:35,948 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:35,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 03:53:35,951 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/table 2023-04-17 03:53:35,952 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 03:53:35,954 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:35,956 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740 2023-04-17 03:53:35,957 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740 2023-04-17 03:53:35,962 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 03:53:35,969 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:53:35,970 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60476744, jitterRate=-0.09882628917694092}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:53:35,970 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 03:53:35,970 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 03:53:35,970 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 03:53:35,971 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 03:53:35,971 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 03:53:35,971 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 03:53:35,972 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 03:53:35,972 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 03:53:35,978 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-17 03:53:35,978 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-17 03:53:35,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-17 03:53:35,998 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-17 03:53:36,000 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-17 03:53:36,016 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43969,1681703612963 with isa=jenkins-hbase4.apache.org/172.31.14.131:36381, startcode=1681703614253 2023-04-17 03:53:36,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,027 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9 2023-04-17 03:53:36,027 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40881 2023-04-17 03:53:36,027 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-17 03:53:36,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 03:53:36,032 DEBUG [RS:0;jenkins-hbase4:36381] zookeeper.ZKUtil(162): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,032 WARN [RS:0;jenkins-hbase4:36381] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-17 03:53:36,032 INFO [RS:0;jenkins-hbase4:36381] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 03:53:36,033 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/WALs/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,034 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36381,1681703614253] 2023-04-17 03:53:36,040 DEBUG [RS:0;jenkins-hbase4:36381] zookeeper.ZKUtil(162): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,050 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-17 03:53:36,057 INFO [RS:0;jenkins-hbase4:36381] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-17 03:53:36,075 INFO [RS:0;jenkins-hbase4:36381] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-17 03:53:36,079 INFO [RS:0;jenkins-hbase4:36381] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-17 03:53:36,079 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,080 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-17 03:53:36,086 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,087 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,087 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,087 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,088 DEBUG [RS:0;jenkins-hbase4:36381] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-17 03:53:36,090 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,090 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,091 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,109 INFO [RS:0;jenkins-hbase4:36381] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-17 03:53:36,112 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36381,1681703614253-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,130 INFO [RS:0;jenkins-hbase4:36381] regionserver.Replication(203): jenkins-hbase4.apache.org,36381,1681703614253 started 2023-04-17 03:53:36,130 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36381,1681703614253, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36381, sessionid=0x101904098950001 2023-04-17 03:53:36,131 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-17 03:53:36,131 DEBUG [RS:0;jenkins-hbase4:36381] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,131 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36381,1681703614253' 2023-04-17 03:53:36,131 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:36,132 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:36,133 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-17 03:53:36,133 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-17 03:53:36,133 DEBUG [RS:0;jenkins-hbase4:36381] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,133 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36381,1681703614253' 2023-04-17 03:53:36,133 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-17 03:53:36,134 DEBUG [RS:0;jenkins-hbase4:36381] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-17 03:53:36,134 DEBUG [RS:0;jenkins-hbase4:36381] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-17 03:53:36,134 INFO [RS:0;jenkins-hbase4:36381] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-17 03:53:36,134 INFO [RS:0;jenkins-hbase4:36381] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-17 03:53:36,152 DEBUG [jenkins-hbase4:43969] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-17 03:53:36,156 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36381,1681703614253, state=OPENING 2023-04-17 03:53:36,164 DEBUG [PEWorker-5] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-17 03:53:36,166 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:36,166 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 03:53:36,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:53:36,245 INFO [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36381%2C1681703614253, suffix=, logDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/WALs/jenkins-hbase4.apache.org,36381,1681703614253, archiveDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/oldWALs, maxLogs=32 2023-04-17 03:53:36,264 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK] 2023-04-17 03:53:36,268 INFO [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/WALs/jenkins-hbase4.apache.org,36381,1681703614253/jenkins-hbase4.apache.org%2C36381%2C1681703614253.1681703616248 2023-04-17 03:53:36,268 DEBUG [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK]] 2023-04-17 03:53:36,355 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,357 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-17 03:53:36,361 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55970, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-17 03:53:36,373 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-17 03:53:36,374 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-17 03:53:36,378 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36381%2C1681703614253.meta, suffix=.meta, logDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/WALs/jenkins-hbase4.apache.org,36381,1681703614253, archiveDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/oldWALs, maxLogs=32 2023-04-17 03:53:36,398 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK] 2023-04-17 03:53:36,402 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/WALs/jenkins-hbase4.apache.org,36381,1681703614253/jenkins-hbase4.apache.org%2C36381%2C1681703614253.meta.1681703616380.meta 2023-04-17 03:53:36,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37359,DS-52013a28-cded-4095-a0f0-389955fa701c,DISK]] 2023-04-17 03:53:36,405 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:53:36,407 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-17 03:53:36,428 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-17 03:53:36,434 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-17 03:53:36,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-17 03:53:36,440 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:36,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-17 03:53:36,441 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-17 03:53:36,444 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-17 03:53:36,446 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/info 2023-04-17 03:53:36,446 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/info 2023-04-17 03:53:36,447 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-17 03:53:36,448 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:36,448 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-17 03:53:36,449 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/rep_barrier 2023-04-17 03:53:36,449 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/rep_barrier 2023-04-17 03:53:36,450 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-17 03:53:36,451 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:36,451 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-17 03:53:36,453 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/table 2023-04-17 03:53:36,453 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/table 2023-04-17 03:53:36,454 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-17 03:53:36,455 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:36,457 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740 2023-04-17 03:53:36,460 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740 2023-04-17 03:53:36,465 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:53:36,467 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-17 03:53:36,469 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68995902, jitterRate=0.028119057416915894}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:53:36,469 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-17 03:53:36,480 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681703616347 2023-04-17 03:53:36,499 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-17 03:53:36,500 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-17 03:53:36,501 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36381,1681703614253, state=OPEN 2023-04-17 03:53:36,505 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-17 03:53:36,505 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-17 03:53:36,509 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-17 03:53:36,510 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36381,1681703614253 in 336 msec 2023-04-17 03:53:36,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-17 03:53:36,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 524 msec 2023-04-17 03:53:36,521 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 805 msec 2023-04-17 03:53:36,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681703616521, completionTime=-1 2023-04-17 03:53:36,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-17 03:53:36,522 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-17 03:53:36,585 DEBUG [hconnection-0x5c683274-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:36,589 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55980, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-17 03:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681703676610 2023-04-17 03:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681703736610 2023-04-17 03:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 88 msec 2023-04-17 03:53:36,633 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43969,1681703612963-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43969,1681703612963-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43969,1681703612963-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,635 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43969, period=300000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,636 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-17 03:53:36,642 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-17 03:53:36,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-17 03:53:36,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-17 03:53:36,662 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-17 03:53:36,665 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:53:36,668 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:53:36,688 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:36,690 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a empty. 2023-04-17 03:53:36,691 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:36,691 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-17 03:53:36,724 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-17 03:53:36,727 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => af5b89de65a7835500f53c143436eb3a, NAME => 'hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:53:36,745 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:36,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing af5b89de65a7835500f53c143436eb3a, disabling compactions & flushes 2023-04-17 03:53:36,746 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:36,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:36,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. after waiting 0 ms 2023-04-17 03:53:36,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:36,746 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:36,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for af5b89de65a7835500f53c143436eb3a: 2023-04-17 03:53:36,751 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:53:36,767 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681703616754"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703616754"}]},"ts":"1681703616754"} 2023-04-17 03:53:36,791 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:53:36,793 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:53:36,799 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703616793"}]},"ts":"1681703616793"} 2023-04-17 03:53:36,802 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-17 03:53:36,812 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=af5b89de65a7835500f53c143436eb3a, ASSIGN}] 2023-04-17 03:53:36,815 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=af5b89de65a7835500f53c143436eb3a, ASSIGN 2023-04-17 03:53:36,817 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=af5b89de65a7835500f53c143436eb3a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:53:36,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=af5b89de65a7835500f53c143436eb3a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:36,968 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681703616967"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703616967"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703616967"}]},"ts":"1681703616967"} 2023-04-17 03:53:36,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure af5b89de65a7835500f53c143436eb3a, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:53:37,137 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:37,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => af5b89de65a7835500f53c143436eb3a, NAME => 'hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:53:37,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:37,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,141 INFO [StoreOpener-af5b89de65a7835500f53c143436eb3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,144 DEBUG [StoreOpener-af5b89de65a7835500f53c143436eb3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/info 2023-04-17 03:53:37,144 DEBUG [StoreOpener-af5b89de65a7835500f53c143436eb3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/info 2023-04-17 03:53:37,144 INFO [StoreOpener-af5b89de65a7835500f53c143436eb3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af5b89de65a7835500f53c143436eb3a columnFamilyName info 2023-04-17 03:53:37,145 INFO [StoreOpener-af5b89de65a7835500f53c143436eb3a-1] regionserver.HStore(310): Store=af5b89de65a7835500f53c143436eb3a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:37,146 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,147 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,151 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for af5b89de65a7835500f53c143436eb3a 2023-04-17 03:53:37,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:53:37,156 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened af5b89de65a7835500f53c143436eb3a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73606970, jitterRate=0.09682932496070862}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-17 03:53:37,156 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for af5b89de65a7835500f53c143436eb3a: 2023-04-17 03:53:37,158 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a., pid=6, masterSystemTime=1681703617128 2023-04-17 03:53:37,163 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:37,163 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:53:37,164 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=af5b89de65a7835500f53c143436eb3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:37,165 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681703617164"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703617164"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703617164"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703617164"}]},"ts":"1681703617164"} 2023-04-17 03:53:37,173 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-17 03:53:37,173 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure af5b89de65a7835500f53c143436eb3a, server=jenkins-hbase4.apache.org,36381,1681703614253 in 195 msec 2023-04-17 03:53:37,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-17 03:53:37,177 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=af5b89de65a7835500f53c143436eb3a, ASSIGN in 361 msec 2023-04-17 03:53:37,178 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:53:37,179 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703617179"}]},"ts":"1681703617179"} 2023-04-17 03:53:37,182 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-17 03:53:37,185 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:53:37,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 533 msec 2023-04-17 03:53:37,265 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-17 03:53:37,267 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-17 03:53:37,267 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:53:37,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-17 03:53:37,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 03:53:37,344 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 39 msec 2023-04-17 03:53:37,356 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-17 03:53:37,372 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-17 03:53:37,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-17 03:53:37,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-17 03:53:37,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-17 03:53:37,397 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.067sec 2023-04-17 03:53:37,399 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-17 03:53:37,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-17 03:53:37,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-17 03:53:37,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43969,1681703612963-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-17 03:53:37,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43969,1681703612963-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-17 03:53:37,412 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-17 03:53:37,503 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:37,507 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:37,520 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:37,531 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:37,542 INFO [Listener at localhost/44831] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:53:37,567 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=169, AvailableMemoryMB=1478 2023-04-17 03:53:37,585 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:53:37,591 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52910, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:53:37,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:53:37,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:53:37,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:53:37,610 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:53:37,613 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:53:37,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-17 03:53:37,618 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:37,621 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef empty. 2023-04-17 03:53:37,623 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:37,623 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:53:37,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 03:53:37,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 03:53:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 03:53:38,043 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:53:38,045 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b6d9193f990a9f1499ce889bc6211eef, NAME => 'TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:53:38,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:38,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b6d9193f990a9f1499ce889bc6211eef, disabling compactions & flushes 2023-04-17 03:53:38,059 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,059 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,059 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. after waiting 0 ms 2023-04-17 03:53:38,059 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,059 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,059 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:38,064 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:53:38,070 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703618070"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703618070"}]},"ts":"1681703618070"} 2023-04-17 03:53:38,074 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:53:38,076 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:53:38,076 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703618076"}]},"ts":"1681703618076"} 2023-04-17 03:53:38,079 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:53:38,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, ASSIGN}] 2023-04-17 03:53:38,085 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, ASSIGN 2023-04-17 03:53:38,086 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:53:38,238 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b6d9193f990a9f1499ce889bc6211eef, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:38,238 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703618238"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703618238"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703618238"}]},"ts":"1681703618238"} 2023-04-17 03:53:38,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 03:53:38,241 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:53:38,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b6d9193f990a9f1499ce889bc6211eef, NAME => 'TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:53:38,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:53:38,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,402 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,404 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,406 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A 2023-04-17 03:53:38,406 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A 2023-04-17 03:53:38,407 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:53:38,408 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6d9193f990a9f1499ce889bc6211eef columnFamilyName A 2023-04-17 03:53:38,408 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(310): Store=b6d9193f990a9f1499ce889bc6211eef/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:38,409 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,410 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B 2023-04-17 03:53:38,411 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B 2023-04-17 03:53:38,411 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:53:38,411 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6d9193f990a9f1499ce889bc6211eef columnFamilyName B 2023-04-17 03:53:38,412 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(310): Store=b6d9193f990a9f1499ce889bc6211eef/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:38,412 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,414 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C 2023-04-17 03:53:38,414 DEBUG [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C 2023-04-17 03:53:38,414 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:53:38,414 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b6d9193f990a9f1499ce889bc6211eef columnFamilyName C 2023-04-17 03:53:38,415 INFO [StoreOpener-b6d9193f990a9f1499ce889bc6211eef-1] regionserver.HStore(310): Store=b6d9193f990a9f1499ce889bc6211eef/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:53:38,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:53:38,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:53:38,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b6d9193f990a9f1499ce889bc6211eef; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66576969, jitterRate=-0.007925853133201599}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:53:38,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:38,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., pid=11, masterSystemTime=1681703618395 2023-04-17 03:53:38,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,431 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,432 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=b6d9193f990a9f1499ce889bc6211eef, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:38,432 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703618432"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703618432"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703618432"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703618432"}]},"ts":"1681703618432"} 2023-04-17 03:53:38,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-17 03:53:38,437 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 in 194 msec 2023-04-17 03:53:38,441 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-17 03:53:38,441 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, ASSIGN in 354 msec 2023-04-17 03:53:38,442 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:53:38,442 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703618442"}]},"ts":"1681703618442"} 2023-04-17 03:53:38,444 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:53:38,448 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:53:38,451 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 842 msec 2023-04-17 03:53:38,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-17 03:53:38,743 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-17 03:53:38,748 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0b17e1c1 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,765 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35594b1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,768 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,776 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:53:38,779 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:53:38,788 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,792 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,793 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,798 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,799 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,803 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,804 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,808 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,809 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,813 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,814 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,820 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,820 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,849 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,849 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,856 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,857 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x4bd6fc59 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:53:38,863 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4808c6d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:53:38,868 DEBUG [hconnection-0x2b3ecf8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,869 DEBUG [hconnection-0x26902f80-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,871 DEBUG [hconnection-0x5ed56db8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,871 DEBUG [hconnection-0x79ea6eef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,871 DEBUG [hconnection-0x38d4eae1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,873 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,873 DEBUG [hconnection-0x38c73e86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,874 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,875 DEBUG [hconnection-0x663bfd7d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,879 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,880 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,880 DEBUG [hconnection-0x513be61b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,885 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,888 DEBUG [hconnection-0x4dbc39ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:53:38,888 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,889 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,888 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:38,895 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:53:38,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:38,919 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:38,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:38,920 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:38,920 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:38,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:38,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:38,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:38,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:38,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:38,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:38,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:38,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:38,927 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:38,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:38,934 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:38,935 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:38,939 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:38,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:38,952 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:38,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:38,954 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:53:38,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:38,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:38,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:38,976 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:38,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:38,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:38,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:38,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:39,106 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703619036, max timestamp=1681703619061Num uniques 859; 2023-04-17 03:53:39,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:53:39,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703619036, max timestamp=1681703619061Num uniques 859; 2023-04-17 03:53:39,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:53:39,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681703619036, max timestamp=1681703619061Num uniques 859; 2023-04-17 03:53:39,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-17 03:53:39,148 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/43410c1d428e41caba2f5ab10837e685 2023-04-17 03:53:39,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703679163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703679164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703679167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703679168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703679176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,298 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/8821ab6a5434412abef56a3b3d3991fa 2023-04-17 03:53:39,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703679287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703679287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703679293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703679294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703679297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,362 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fc0fadbe51a34ba6a002e54a293eb574 2023-04-17 03:53:39,375 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/43410c1d428e41caba2f5ab10837e685 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685 2023-04-17 03:53:39,387 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685, entries=150, sequenceid=11, filesize=11.6 K 2023-04-17 03:53:39,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/8821ab6a5434412abef56a3b3d3991fa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa 2023-04-17 03:53:39,422 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa, entries=150, sequenceid=11, filesize=11.6 K 2023-04-17 03:53:39,424 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fc0fadbe51a34ba6a002e54a293eb574 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574 2023-04-17 03:53:39,436 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574, entries=150, sequenceid=11, filesize=11.6 K 2023-04-17 03:53:39,441 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.26 KB/141575 for b6d9193f990a9f1499ce889bc6211eef in 487ms, sequenceid=11, compaction requested=false 2023-04-17 03:53:39,445 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-17 03:53:39,448 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:39,448 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:39,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:39,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:39,452 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:39,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:39,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dbb4e4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:39,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:39,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:39,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:39,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:39,464 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:39,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:39,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:39,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:39,484 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:39,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:39,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:39,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:39,484 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:39,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:39,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:39,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:39,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:39,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:39,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:39,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:39,521 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:39,521 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:39,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:39,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=144.97 KB heapSize=414.85 KB 2023-04-17 03:53:39,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703619154, max timestamp=1681703619527Num uniques 1068; 2023-04-17 03:53:39,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703619154, max timestamp=1681703619527Num uniques 1068; 2023-04-17 03:53:39,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 03:53:39,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-17 03:53:39,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681703619154, max timestamp=1681703619527Num uniques 1009; 2023-04-17 03:53:39,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-17 03:53:39,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:39,566 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:39,566 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:39,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:39,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:39,567 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:39,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:39,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:39,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/251bf04cb37c47d28797b78e47f09950 2023-04-17 03:53:39,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:39,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:39,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:39,713 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:39,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:39,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:39,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:39,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,730 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:39,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:39,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703619674, max timestamp=1681703619696Num uniques 859; 2023-04-17 03:53:39,732 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:39,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:39,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:39,734 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:39,734 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:39,734 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:39,735 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:39,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,735 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:39,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:39,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703619674, max timestamp=1681703619696Num uniques 859; 2023-04-17 03:53:39,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:39,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703619674, max timestamp=1681703619697Num uniques 859; 2023-04-17 03:53:39,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:39,735 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:39,742 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:39,742 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:39,745 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:39,745 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:39,745 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,745 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:39,745 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:39,751 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,752 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:39,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:39,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2049db[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:39,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:39,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,767 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:39,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:39,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:39,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:39,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:39,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:39,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:39,772 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:39,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:39,779 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:39,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:39,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:39,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:39,779 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:39,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:39,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,788 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:39,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:39,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:39,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:39,806 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:39,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:39,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:39,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:39,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:39,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:39,808 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:39,808 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:39,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,808 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,808 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,809 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,809 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,809 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703679823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:39,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703679827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703619819, max timestamp=1681703619830Num uniques 1018; 2023-04-17 03:53:39,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:53:39,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703619819, max timestamp=1681703619830Num uniques 1018; 2023-04-17 03:53:39,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:53:39,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681703619819, max timestamp=1681703619826Num uniques 1018; 2023-04-17 03:53:39,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-17 03:53:39,908 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:39,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:39,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:39,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703619891, max timestamp=1681703619917Num uniques 1009; 2023-04-17 03:53:39,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12682, suffix heap size=132904,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=277 2023-04-17 03:53:39,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703619891, max timestamp=1681703619917Num uniques 1027; 2023-04-17 03:53:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:39,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:53:39,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:39,924 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:39,924 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:39,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703619891, max timestamp=1681703619917Num uniques 1027; 2023-04-17 03:53:39,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:39,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:39,926 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,926 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:53:39,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:39,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:39,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:39,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:39,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:39,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:39,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:39,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:39,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:39,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:39,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:39,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:39,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:39,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:39,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:39,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:39,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:39,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:39,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:39,979 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:39,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dd6cde6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:39,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:39,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:39,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:39,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:39,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:39,988 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:39,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:39,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:39,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:39,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:39,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:39,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:39,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:40,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703619972, max timestamp=1681703619996Num uniques 1136; 2023-04-17 03:53:40,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 03:53:40,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:40,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:40,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:40,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:40,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703619972, max timestamp=1681703619996Num uniques 1136; 2023-04-17 03:53:40,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 03:53:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:40,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:40,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:40,005 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703619972, max timestamp=1681703619996Num uniques 1136; 2023-04-17 03:53:40,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-17 03:53:40,006 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:40,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:40,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:40,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:40,021 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:40,021 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:40,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:40,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,021 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,021 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703680044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703680048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681703620033, max timestamp=1681703620057Num uniques 1095; 2023-04-17 03:53:40,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-17 03:53:40,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703680048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703620033, max timestamp=1681703620057Num uniques 1095; 2023-04-17 03:53:40,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703620033, max timestamp=1681703620057Num uniques 1095; 2023-04-17 03:53:40,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:53:40,069 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:53:40,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:40,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:40,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703680126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703680127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:40,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:40,140 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:40,140 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:40,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:40,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:40,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:40,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:40,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:40,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:40,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:40,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:40,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:40,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:40,153 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:40,153 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:40,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:40,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:40,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:40,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:40,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:40,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:40,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:40,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:40,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:40,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a78ce2c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:40,165 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:40,165 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,166 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/5f0b782bcc3b4877a9a50d7e98e3c65b 2023-04-17 03:53:40,168 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:40,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:40,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:40,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703680165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703680167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703680173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,178 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:40,178 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:40,178 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:40,180 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:40,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:40,181 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:40,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:40,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:40,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:40,182 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:40,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:40,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:40,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:40,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,185 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:40,188 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:40,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:40,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:40,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:40,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:40,199 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:40,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:40,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:40,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:40,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:40,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:40,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=12.62 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/9c04d32fdb7c4226869556e774bc156b 2023-04-17 03:53:40,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703680245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703680245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/251bf04cb37c47d28797b78e47f09950 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950 2023-04-17 03:53:40,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950, entries=150, sequenceid=51, filesize=11.6 K 2023-04-17 03:53:40,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/5f0b782bcc3b4877a9a50d7e98e3c65b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b 2023-04-17 03:53:40,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-17 03:53:40,300 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:40,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/9c04d32fdb7c4226869556e774bc156b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b 2023-04-17 03:53:40,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:40,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b, entries=150, sequenceid=51, filesize=11.6 K 2023-04-17 03:53:40,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:40,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~39.46 KB/40402, heapSize ~103.36 KB/105840, currentSize=149.83 KB/153430 for b6d9193f990a9f1499ce889bc6211eef in 777ms, sequenceid=51, compaction requested=false 2023-04-17 03:53:40,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:40,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:40,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:40,323 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:40,323 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:40,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:40,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:40,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,326 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:40,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:40,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:40,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:40,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:40,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:40,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:40,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:40,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:40,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:40,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:40,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:40,344 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=149.83 KB heapSize=417.16 KB 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:40,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:40,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:40,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703680399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703680405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703680404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703680453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703680453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703680507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703680508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703680510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703680710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703680711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703680712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703680760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:40,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703680760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:40,801 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/b3cfd282401f42cfaeca6201a52aaaa7 2023-04-17 03:53:41,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703681014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703681016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703681019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703681266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703681275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,282 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/1c2a41cff28141309ad4c7dc67616180 2023-04-17 03:53:41,361 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/0bf994dfcf05429694360acbda6ef11a 2023-04-17 03:53:41,374 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/b3cfd282401f42cfaeca6201a52aaaa7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7 2023-04-17 03:53:41,386 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7, entries=150, sequenceid=156, filesize=11.8 K 2023-04-17 03:53:41,390 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/1c2a41cff28141309ad4c7dc67616180 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180 2023-04-17 03:53:41,406 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180, entries=150, sequenceid=156, filesize=11.8 K 2023-04-17 03:53:41,409 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/0bf994dfcf05429694360acbda6ef11a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a 2023-04-17 03:53:41,422 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a, entries=150, sequenceid=156, filesize=11.8 K 2023-04-17 03:53:41,425 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~417.11 KB/427120, currentSize=33.54 KB/34350 for b6d9193f990a9f1499ce889bc6211eef in 1079ms, sequenceid=156, compaction requested=true 2023-04-17 03:53:41,425 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:41,425 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,425 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:41,436 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:41,437 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:41,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:41,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6082713[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:41,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:41,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:41,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:41,452 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:41,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:41,466 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:41,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:41,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:41,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:41,466 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:41,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,482 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:41,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:41,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,497 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,497 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,496 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:41,498 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:41,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:41,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:41,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:41,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:41,599 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:41,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:41,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:41,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703621584, max timestamp=1681703621604Num uniques 859; 2023-04-17 03:53:41,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:53:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703621584, max timestamp=1681703621604Num uniques 859; 2023-04-17 03:53:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:53:41,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703621584, max timestamp=1681703621604Num uniques 859; 2023-04-17 03:53:41,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:53:41,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:41,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:41,616 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:41,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,617 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:41,617 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:41,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,619 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:41,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:41,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:41,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:41,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:41,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:41,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:41,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:41,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:41,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:41,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:41,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:41,627 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,627 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:41,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:41,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eced898[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:41,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:41,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,638 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:41,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:41,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:41,639 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,639 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:41,640 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:41,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:41,648 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:41,648 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:41,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:41,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:41,649 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:41,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703681643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703681646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703681649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,664 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:41,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:41,669 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:41,669 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:41,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703681751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703681752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703681754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,769 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:41,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:41,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:41,777 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:41,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:41,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:41,782 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,782 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:41,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:41,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:41,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:41,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:41,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:41,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:41,791 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,791 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:41,791 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:41,792 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:41,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:41,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:41,796 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,797 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,798 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,798 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:41,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:41,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550c895a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:41,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:41,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:41,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:41,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:41,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:41,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:41,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:41,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:41,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:41,814 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:41,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:41,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:41,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:41,814 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:41,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,824 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:41,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:41,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:41,835 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,835 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:41,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,935 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:41,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:41,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:41,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:41,945 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:41,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:41,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:41,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:41,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:41,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:41,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:41,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:41,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:41,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:41,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:41,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:41,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:41,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:41,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:41,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:41,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:41,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,960 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:41,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703681958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:41,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c93479[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:41,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:41,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703681959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:41,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703681962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:41,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:41,972 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:41,972 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:41,974 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,974 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:41,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:41,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,981 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:41,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/dccc34921c034d0aa9d0fb0f48c023b6 2023-04-17 03:53:41,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:41,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:41,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:41,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:41,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:41,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:41,990 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:41,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:42,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:41,997 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,010 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:42,010 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:42,023 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:53:42,111 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:42,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:42,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:42,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:42,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:42,125 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:42,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:42,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:42,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/f54ceb9bd3d74e1eb651fd1be43bb3ef 2023-04-17 03:53:42,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,130 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:42,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:42,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:42,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:42,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:42,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:42,153 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:42,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:42,154 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:42,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:42,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:42,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebeb8ec[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:42,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:42,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,180 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:42,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:42,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:42,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:42,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:42,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:42,195 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:42,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:42,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:42,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:42,196 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:42,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/8de6966b3ebc40469ddf5e3f280d5c76 2023-04-17 03:53:42,205 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,205 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,206 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/dccc34921c034d0aa9d0fb0f48c023b6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6 2023-04-17 03:53:42,215 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-17 03:53:42,217 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-17 03:53:42,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:42,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:42,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,219 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:42,219 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:42,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6, entries=250, sequenceid=166, filesize=16.4 K 2023-04-17 03:53:42,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/f54ceb9bd3d74e1eb651fd1be43bb3ef as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef 2023-04-17 03:53:42,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef, entries=150, sequenceid=166, filesize=11.8 K 2023-04-17 03:53:42,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/8de6966b3ebc40469ddf5e3f280d5c76 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76 2023-04-17 03:53:42,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703682266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703682268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703682269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76, entries=150, sequenceid=166, filesize=11.8 K 2023-04-17 03:53:42,289 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b6d9193f990a9f1499ce889bc6211eef in 758ms, sequenceid=166, compaction requested=true 2023-04-17 03:53:42,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:42,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:42,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:42,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:42,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:53:42,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-17 03:53:42,299 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:42,299 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:42,299 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:42,299 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:42,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:42,300 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:42,300 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:42,300 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:42,317 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47844 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:42,319 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:42,320 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:42,320 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:42,320 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=46.7 K 2023-04-17 03:53:42,322 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8821ab6a5434412abef56a3b3d3991fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681703618909 2023-04-17 03:53:42,323 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5f0b782bcc3b4877a9a50d7e98e3c65b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681703619165 2023-04-17 03:53:42,323 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1c2a41cff28141309ad4c7dc67616180, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681703620106 2023-04-17 03:53:42,324 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f54ceb9bd3d74e1eb651fd1be43bb3ef, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620401 2023-04-17 03:53:42,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:42,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703682336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703682337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:42,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:42,348 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:42,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,348 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52624 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:42,348 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:42,349 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,349 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=51.4 K 2023-04-17 03:53:42,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 43410c1d428e41caba2f5ab10837e685, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681703618909 2023-04-17 03:53:42,350 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 251bf04cb37c47d28797b78e47f09950, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681703619165 2023-04-17 03:53:42,353 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:42,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:42,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,357 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b3cfd282401f42cfaeca6201a52aaaa7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681703620106 2023-04-17 03:53:42,357 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,357 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting dccc34921c034d0aa9d0fb0f48c023b6, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620378 2023-04-17 03:53:42,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:42,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:42,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:42,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:42,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:42,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:42,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:42,375 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:42,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:42,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:42,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:42,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9b8d0a6f1f6c4beb9edbca27b916e218 2023-04-17 03:53:42,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,382 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:42,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ca1bb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:42,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:42,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,388 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:42,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:42,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:42,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:42,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:42,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:42,396 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:42,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:42,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:42,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:42,396 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:42,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,406 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:42,413 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:42,414 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:42,414 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:42,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,416 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#14 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:42,416 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#13 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:42,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703682442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703682449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,515 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:42,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/2bf2b93b0c6f49a3966d89a3e0e72f91 2023-04-17 03:53:42,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:42,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:42,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:42,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:42,567 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:42,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:42,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:42,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:42,571 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:42,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:42,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,581 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/21e7970f9a8646248c8809c3574eef80 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/21e7970f9a8646248c8809c3574eef80 2023-04-17 03:53:42,584 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:42,598 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/af4a2d262195451c919e6ec000385326 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af4a2d262195451c919e6ec000385326 2023-04-17 03:53:42,602 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,602 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:42,602 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:42,605 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:42,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:42,618 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379bae75[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:42,622 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:42,622 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,623 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:42,623 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,623 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into 21e7970f9a8646248c8809c3574eef80(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:42,624 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:42,624 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:42,624 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=12, startTime=1681703622298; duration=0sec 2023-04-17 03:53:42,623 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into af4a2d262195451c919e6ec000385326(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:42,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:42,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:42,624 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:42,624 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=12, startTime=1681703622291; duration=0sec 2023-04-17 03:53:42,629 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:42,629 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:42,630 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:42,653 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47844 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:42,653 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:42,653 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,653 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=46.7 K 2023-04-17 03:53:42,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,657 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:42,659 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fc0fadbe51a34ba6a002e54a293eb574, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1681703618909 2023-04-17 03:53:42,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,661 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:42,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,661 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9c04d32fdb7c4226869556e774bc156b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681703619165 2023-04-17 03:53:42,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,667 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0bf994dfcf05429694360acbda6ef11a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681703620106 2023-04-17 03:53:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,668 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8de6966b3ebc40469ddf5e3f280d5c76, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620401 2023-04-17 03:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:42,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:42,671 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:42,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:42,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:42,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:42,671 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:42,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,679 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703682663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703682666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/ef53a3d7281344cf8182c17c85eab2c4 2023-04-17 03:53:42,697 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,697 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,697 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,697 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:42,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:42,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:42,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:42,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9b8d0a6f1f6c4beb9edbca27b916e218 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218 2023-04-17 03:53:42,724 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#17 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:42,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218, entries=250, sequenceid=203, filesize=16.4 K 2023-04-17 03:53:42,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/2bf2b93b0c6f49a3966d89a3e0e72f91 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91 2023-04-17 03:53:42,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91, entries=150, sequenceid=203, filesize=11.8 K 2023-04-17 03:53:42,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:42,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:42,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703682791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703682794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:42,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703682791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,805 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/ef53a3d7281344cf8182c17c85eab2c4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4 2023-04-17 03:53:42,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:42,809 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:42,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,812 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,813 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:42,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:42,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:42,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:42,817 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:42,819 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:42,819 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,820 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:42,820 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:42,820 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:42,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:42,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4, entries=150, sequenceid=203, filesize=11.8 K 2023-04-17 03:53:42,826 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for b6d9193f990a9f1499ce889bc6211eef in 527ms, sequenceid=203, compaction requested=false 2023-04-17 03:53:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:42,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a13e03[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:42,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:42,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:42,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:42,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:42,844 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:42,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:42,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:42,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:42,851 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:42,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:42,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:42,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:42,851 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:42,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,860 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:42,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:42,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:42,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:42,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:42,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:42,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:42,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:42,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:42,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:42,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:42,983 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:42,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:42,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:42,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,987 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:42,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:42,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:42,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:42,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:42,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:42,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:42,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:42,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:42,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:42,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:42,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:42,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:42,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:43,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:43,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:43,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:43,002 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:43,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7393c3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:43,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:43,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,010 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:43,010 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:43,010 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:43,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:43,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:43,013 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:43,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:43,019 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:43,019 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:43,030 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,030 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,037 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:43,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:43,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:43,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:43,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:43,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/290bfe7855d84458a8fa31bd4e955ebc 2023-04-17 03:53:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703623036, max timestamp=1681703623080Num uniques 859; 2023-04-17 03:53:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703623036, max timestamp=1681703623080Num uniques 859; 2023-04-17 03:53:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:43,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703623036, max timestamp=1681703623080Num uniques 859; 2023-04-17 03:53:43,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:43,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/da2d0ca0814f4efb9246721ec38d6cf5 2023-04-17 03:53:43,133 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 03:53:43,133 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-17 03:53:43,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:43,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:43,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:43,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703683147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703683155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:43,158 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:43,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:43,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:43,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,166 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,166 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:43,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:43,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:43,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:43,183 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:43,183 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:43,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:43,187 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,188 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,188 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff1928f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:43,192 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:43,192 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,193 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:43,194 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:43,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:43,196 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,217 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/109748810eb141d899bd27c456e46aa9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/109748810eb141d899bd27c456e46aa9 2023-04-17 03:53:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:43,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:43,222 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:43,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:43,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:43,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:43,223 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:43,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:43,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,242 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:43,242 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:43,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,242 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 109748810eb141d899bd27c456e46aa9(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:43,243 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:43,243 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=12, startTime=1681703622298; duration=0sec 2023-04-17 03:53:43,243 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:43,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703683259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703683259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,343 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:43,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:43,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:43,358 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:43,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:43,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:43,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,363 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:43,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:43,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:43,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:43,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:43,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:43,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:43,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:43,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:43,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:43,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f37902[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:43,383 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:43,383 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,389 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,389 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:43,389 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:43,389 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:43,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:43,391 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:43,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:43,397 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:43,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:43,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:43,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:43,397 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:43,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,406 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,410 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:43,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:43,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:43,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:43,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703683467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703683473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:43,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:43,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:43,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:43,533 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:43,533 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:43,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:43,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,535 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,537 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:43,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:43,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:43,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:43,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:43,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:43,541 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:43,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:43,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:43,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:43,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:43,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:43,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:43,546 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:43,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bcc1699[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:43,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:43,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,551 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:43,551 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,551 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:43,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:43,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:43,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:43,560 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:43,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:43,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:43,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:43,560 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:43,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,565 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,573 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:43,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:43,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:43,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,574 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:43,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:43,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/7c5bac05ad93416dbb22eabac0a636fe 2023-04-17 03:53:43,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/290bfe7855d84458a8fa31bd4e955ebc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc 2023-04-17 03:53:43,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc, entries=200, sequenceid=215, filesize=14.1 K 2023-04-17 03:53:43,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/da2d0ca0814f4efb9246721ec38d6cf5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5 2023-04-17 03:53:43,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5, entries=150, sequenceid=215, filesize=11.8 K 2023-04-17 03:53:43,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/7c5bac05ad93416dbb22eabac0a636fe as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe 2023-04-17 03:53:43,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe, entries=150, sequenceid=215, filesize=11.8 K 2023-04-17 03:53:43,642 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b6d9193f990a9f1499ce889bc6211eef in 649ms, sequenceid=215, compaction requested=true 2023-04-17 03:53:43,642 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:43,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:43,642 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:53:43,642 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:53:43,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:43,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:43,646 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:53:43,646 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:53:43,646 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:43,647 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:43,647 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,647 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af4a2d262195451c919e6ec000385326, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=42.4 K 2023-04-17 03:53:43,647 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,647 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/21e7970f9a8646248c8809c3574eef80, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=35.4 K 2023-04-17 03:53:43,649 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting af4a2d262195451c919e6ec000385326, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620401 2023-04-17 03:53:43,653 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 21e7970f9a8646248c8809c3574eef80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620401 2023-04-17 03:53:43,654 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9b8d0a6f1f6c4beb9edbca27b916e218, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681703621638 2023-04-17 03:53:43,654 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2bf2b93b0c6f49a3966d89a3e0e72f91, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681703621645 2023-04-17 03:53:43,655 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 290bfe7855d84458a8fa31bd4e955ebc, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622303 2023-04-17 03:53:43,655 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting da2d0ca0814f4efb9246721ec38d6cf5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622335 2023-04-17 03:53:43,675 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:43,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:43,688 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:43,688 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:43,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:43,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:43,695 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:43,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:43,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:43,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,707 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,710 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,710 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:43,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:43,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:43,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:43,728 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:43,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:43,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:43,729 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 03:53:43,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:43,730 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:43,730 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:43,730 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:43,730 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:43,730 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:43,736 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/8234740a35b541f2a325d6b53851aa04 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/8234740a35b541f2a325d6b53851aa04 2023-04-17 03:53:43,756 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/af6b921e082c4806b319156cb6c37e81 2023-04-17 03:53:43,757 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into 8234740a35b541f2a325d6b53851aa04(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:43,758 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:43,758 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=13, startTime=1681703623642; duration=0sec 2023-04-17 03:53:43,758 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:43,758 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:53:43,761 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:53:43,761 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:43,761 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,761 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/109748810eb141d899bd27c456e46aa9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=35.4 K 2023-04-17 03:53:43,762 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 109748810eb141d899bd27c456e46aa9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703620401 2023-04-17 03:53:43,762 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ef53a3d7281344cf8182c17c85eab2c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1681703621645 2023-04-17 03:53:43,763 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 7c5bac05ad93416dbb22eabac0a636fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622335 2023-04-17 03:53:43,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:43,780 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:43,790 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#25 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:43,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703683791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703683797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703683804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703683807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703683813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,827 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/8989136176f742febb59058cd0691f4b 2023-04-17 03:53:43,850 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/108e129382f54eafad1d34c0546833ce as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/108e129382f54eafad1d34c0546833ce 2023-04-17 03:53:43,863 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1d4bd20858f544c893969c8cd9dd56c6 2023-04-17 03:53:43,867 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 108e129382f54eafad1d34c0546833ce(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:43,868 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:43,868 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=13, startTime=1681703623642; duration=0sec 2023-04-17 03:53:43,868 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:43,879 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/af6b921e082c4806b319156cb6c37e81 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81 2023-04-17 03:53:43,886 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 03:53:43,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/8989136176f742febb59058cd0691f4b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b 2023-04-17 03:53:43,915 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 03:53:43,918 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1d4bd20858f544c893969c8cd9dd56c6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6 2023-04-17 03:53:43,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:43,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703683915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703683915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,936 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6, entries=150, sequenceid=253, filesize=11.8 K 2023-04-17 03:53:43,938 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for b6d9193f990a9f1499ce889bc6211eef in 209ms, sequenceid=253, compaction requested=false 2023-04-17 03:53:43,938 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:43,938 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:43,938 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:43,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:43,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:43,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7165a8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:43,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:43,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:43,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:43,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:43,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:43,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:43,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:43,954 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:43,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:43,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:43,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:43,955 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:43,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:43,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:43,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:43,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:43,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:43,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:43,986 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:43,986 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,986 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:43,986 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:43,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:43,986 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:43,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:43,986 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:43,987 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:43,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:44,087 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:44,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:44,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:44,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:44,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:44,093 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:44,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:44,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:44,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:44,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:44,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:44,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:44,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:44,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:44,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:44,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:44,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:44,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:44,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:44,102 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:44,103 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:44,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:53:44,104 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:44,105 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:44,105 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:44,105 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:44,105 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:44,105 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:44,143 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/f05386bac67147629a1f18b17a91cfba as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f05386bac67147629a1f18b17a91cfba 2023-04-17 03:53:44,145 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/04226128fac34f95aa9a10c2a0ff3213 2023-04-17 03:53:44,146 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:44,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:44,159 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into f05386bac67147629a1f18b17a91cfba(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:44,159 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:44,159 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=13, startTime=1681703623642; duration=0sec 2023-04-17 03:53:44,159 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:44,181 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/c6bcacf514c74572becd4b0bb37a3a80 2023-04-17 03:53:44,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703624190, max timestamp=1681703624211Num uniques 859; 2023-04-17 03:53:44,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703624190, max timestamp=1681703624211Num uniques 859; 2023-04-17 03:53:44,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:53:44,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:53:44,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703624190, max timestamp=1681703624211Num uniques 859; 2023-04-17 03:53:44,215 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:44,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703684255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703684255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703684363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703684363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703684568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703684569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,609 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/29280b30901c4db9a3be2f579904337c 2023-04-17 03:53:44,616 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/04226128fac34f95aa9a10c2a0ff3213 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213 2023-04-17 03:53:44,623 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213, entries=150, sequenceid=265, filesize=11.9 K 2023-04-17 03:53:44,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/c6bcacf514c74572becd4b0bb37a3a80 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80 2023-04-17 03:53:44,632 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80, entries=150, sequenceid=265, filesize=11.9 K 2023-04-17 03:53:44,633 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/29280b30901c4db9a3be2f579904337c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c 2023-04-17 03:53:44,645 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c, entries=150, sequenceid=265, filesize=11.9 K 2023-04-17 03:53:44,647 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=143.13 KB/146560 for b6d9193f990a9f1499ce889bc6211eef in 544ms, sequenceid=265, compaction requested=true 2023-04-17 03:53:44,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:44,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:44,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:44,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:44,650 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:44,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:44,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58b7f5a3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:44,653 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:44,653 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:44,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:44,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:44,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:44,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:44,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:44,668 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:44,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:44,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:44,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:44,668 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:44,673 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,673 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:44,673 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:44,674 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:44,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:44,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:44,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:44,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,682 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,682 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,682 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:44,682 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:44,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:44,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:44,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:44,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:44,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:44,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:44,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:44,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:44,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:44,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:44,791 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:44,791 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:44,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:44,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:44,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,795 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:44,795 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:44,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:44,796 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,796 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:44,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:44,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:44,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:44,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:44,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:44,799 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:44,805 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:44,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:44,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:44,815 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:44,827 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:44,827 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:44,827 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=143.13 KB heapSize=405.30 KB 2023-04-17 03:53:44,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:44,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:44,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:44,846 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/1f7349c58efc423d950264ed60629d6f 2023-04-17 03:53:44,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:44,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:44,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703684899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:44,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:44,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703684901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703685007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703685007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703685218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703685219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,319 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/299e5e554ef043a0bdde8aad7fdf1699 2023-04-17 03:53:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56044 deadline: 1681703685523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56024 deadline: 1681703685526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,782 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/2beb89fa786f4b9cbe8413dc544b208a 2023-04-17 03:53:45,790 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/1f7349c58efc423d950264ed60629d6f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f 2023-04-17 03:53:45,798 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 03:53:45,802 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/299e5e554ef043a0bdde8aad7fdf1699 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699 2023-04-17 03:53:45,813 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 03:53:45,815 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/2beb89fa786f4b9cbe8413dc544b208a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a 2023-04-17 03:53:45,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703685822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,825 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:45,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703685823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:45,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703685823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,832 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:45,833 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 03:53:45,836 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:45,838 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~405.25 KB/414976, currentSize=40.25 KB/41220 for b6d9193f990a9f1499ce889bc6211eef in 1011ms, sequenceid=303, compaction requested=true 2023-04-17 03:53:45,839 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:45,839 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:45,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2acde[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:45,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:45,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,848 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:45,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:45,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:45,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:45,854 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:45,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:45,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:45,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:45,854 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:45,856 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,856 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:45,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,857 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:45,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:45,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:45,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:45,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:45,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:45,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:45,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:45,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:45,882 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:45,882 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:45,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:45,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:45,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:45,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:45,982 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:45,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:45,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:45,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:45,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:45,987 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:45,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:45,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:45,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:45,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:45,996 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:45,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:45,996 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:45,997 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:45,997 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:45,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:45,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:45,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:45,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:46,000 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,000 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:46,001 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:46,001 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:46,025 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/91d6d9fca5214c54aaab457d98ffb9e8 2023-04-17 03:53:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:46,040 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:46,051 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/b07ee0166cfd41638ad686cb89f9c365 2023-04-17 03:53:46,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703626079, max timestamp=1681703626092Num uniques 859; 2023-04-17 03:53:46,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703626079, max timestamp=1681703626092Num uniques 859; 2023-04-17 03:53:46,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:46,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703626079, max timestamp=1681703626092Num uniques 859; 2023-04-17 03:53:46,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:46,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:46,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703686137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703626124, max timestamp=1681703626137Num uniques 1018; 2023-04-17 03:53:46,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703626124, max timestamp=1681703626137Num uniques 1018; 2023-04-17 03:53:46,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:53:46,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703626124, max timestamp=1681703626137Num uniques 1018; 2023-04-17 03:53:46,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:53:46,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:53:46,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703686227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703686243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703686332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703686450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,471 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/e9dcd7be1e764bb4988f0e9675951751 2023-04-17 03:53:46,481 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/91d6d9fca5214c54aaab457d98ffb9e8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8 2023-04-17 03:53:46,493 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8, entries=100, sequenceid=312, filesize=9.5 K 2023-04-17 03:53:46,500 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/b07ee0166cfd41638ad686cb89f9c365 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365 2023-04-17 03:53:46,513 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365, entries=100, sequenceid=312, filesize=9.5 K 2023-04-17 03:53:46,514 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/e9dcd7be1e764bb4988f0e9675951751 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751 2023-04-17 03:53:46,529 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751, entries=100, sequenceid=312, filesize=9.5 K 2023-04-17 03:53:46,530 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for b6d9193f990a9f1499ce889bc6211eef in 529ms, sequenceid=312, compaction requested=true 2023-04-17 03:53:46,530 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:46,530 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,530 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:46,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:46,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:46,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:46,536 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,536 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:46,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d610a43[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:46,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:46,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,541 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:46,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:46,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:46,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=147.60 KB heapSize=420 KB 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:46,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:53:46,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:46,546 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:46,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:46,554 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:46,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:46,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:46,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:46,554 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:46,556 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,556 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,556 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,556 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,573 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:46,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:46,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,573 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:46,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703686577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:46,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:46,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:46,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:46,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:46,687 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:46,687 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,687 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:46,687 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:46,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,691 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,691 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:46,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:46,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:46,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703686686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:46,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:46,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:46,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:46,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:46,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:46,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4d6a62[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:46,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:46,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:46,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:46,703 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:46,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:46,706 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:46,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:46,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:46,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:46,706 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:46,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,710 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,712 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:46,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:46,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:46,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:46,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703686759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:46,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:46,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:46,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:46,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:46,827 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:46,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:46,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:46,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:46,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:46,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:46,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:46,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:46,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:46,832 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab08a85[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:46,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:46,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:46,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:46,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:46,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:46,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:46,841 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:46,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:46,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:46,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:46,841 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:46,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:46,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,847 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:46,848 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:46,848 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:46,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:46,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:46,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703686896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,948 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:46,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:46,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:46,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:46,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:46,968 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:46,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:46,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:46,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:46,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:46,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:46,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:46,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:46,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:46,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:46,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:46,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:46,975 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:46,980 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:46,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49976951[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:46,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:46,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/abae228356174384bb575a448b6fa08b 2023-04-17 03:53:46,990 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:46,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:46,992 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:46,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:46,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:46,998 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:46,998 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:46,998 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:46,998 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:46,998 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:47,000 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,000 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:47,000 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:47,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:47,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,004 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:47,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:47,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:47,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:47,009 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:47,009 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:47,011 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:47,011 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:47,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:47,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:47,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/6c1eefdfeb0b4506b86969046ec82210 2023-04-17 03:53:47,068 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/2dc665b86a934a6bbd3ad1aadd8f981e 2023-04-17 03:53:47,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/abae228356174384bb575a448b6fa08b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b 2023-04-17 03:53:47,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b, entries=200, sequenceid=367, filesize=14.3 K 2023-04-17 03:53:47,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/6c1eefdfeb0b4506b86969046ec82210 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210 2023-04-17 03:53:47,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 03:53:47,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/2dc665b86a934a6bbd3ad1aadd8f981e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e 2023-04-17 03:53:47,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 03:53:47,109 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:47,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:47,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.95 KB/430032, currentSize=33.54 KB/34350 for b6d9193f990a9f1499ce889bc6211eef in 567ms, sequenceid=367, compaction requested=true 2023-04-17 03:53:47,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:47,111 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:47,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:47,112 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:47,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:47,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:47,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:47,116 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73052 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:47,116 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:47,116 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:47,116 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/8234740a35b541f2a325d6b53851aa04, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=71.3 K 2023-04-17 03:53:47,117 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8234740a35b541f2a325d6b53851aa04, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622335 2023-04-17 03:53:47,118 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70612 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:47,119 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting af6b921e082c4806b319156cb6c37e81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681703623137 2023-04-17 03:53:47,129 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:47,129 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:47,129 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f05386bac67147629a1f18b17a91cfba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=69.0 K 2023-04-17 03:53:47,129 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 04226128fac34f95aa9a10c2a0ff3213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681703623786 2023-04-17 03:53:47,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:47,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:47,131 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f05386bac67147629a1f18b17a91cfba, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622335 2023-04-17 03:53:47,131 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:47,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:47,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:47,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:47,132 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1f7349c58efc423d950264ed60629d6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703624247 2023-04-17 03:53:47,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:47,132 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8989136176f742febb59058cd0691f4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681703623137 2023-04-17 03:53:47,132 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 91d6d9fca5214c54aaab457d98ffb9e8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681703624885 2023-04-17 03:53:47,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c6bcacf514c74572becd4b0bb37a3a80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681703623786 2023-04-17 03:53:47,133 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,133 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting abae228356174384bb575a448b6fa08b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:47,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 299e5e554ef043a0bdde8aad7fdf1699, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703624247 2023-04-17 03:53:47,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:47,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:47,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:47,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:47,135 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b07ee0166cfd41638ad686cb89f9c365, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681703624885 2023-04-17 03:53:47,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:47,137 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 6c1eefdfeb0b4506b86969046ec82210, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:47,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:47,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:47,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:47,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:47,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:47,145 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-17 03:53:47,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:47,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:47,147 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:47,147 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:47,147 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:47,147 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:47,166 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#41 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:47,168 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#40 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:47,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:47,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703627247, max timestamp=1681703627272Num uniques 859; 2023-04-17 03:53:47,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:47,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703627247, max timestamp=1681703627272Num uniques 859; 2023-04-17 03:53:47,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:47,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703627247, max timestamp=1681703627272Num uniques 859; 2023-04-17 03:53:47,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:47,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703627308, max timestamp=1681703627328Num uniques 1018; 2023-04-17 03:53:47,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:53:47,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703627308, max timestamp=1681703627328Num uniques 1018; 2023-04-17 03:53:47,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:53:47,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703627308, max timestamp=1681703627328Num uniques 1018; 2023-04-17 03:53:47,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:53:47,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703627390, max timestamp=1681703627405Num uniques 1077; 2023-04-17 03:53:47,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703627390, max timestamp=1681703627405Num uniques 1027; 2023-04-17 03:53:47,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703627390, max timestamp=1681703627405Num uniques 1027; 2023-04-17 03:53:47,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:53:47,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-17 03:53:47,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:53:47,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703687452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703687453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703687563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703687563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,585 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/71c1eb6d91244f06a33bd27340c7db4c 2023-04-17 03:53:47,595 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/16eeca9cb729447ab932c7b90eca0793 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/16eeca9cb729447ab932c7b90eca0793 2023-04-17 03:53:47,614 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/ec9f51ce81ea4ad19bc57634a90f8058 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ec9f51ce81ea4ad19bc57634a90f8058 2023-04-17 03:53:47,622 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into 16eeca9cb729447ab932c7b90eca0793(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:47,622 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:47,622 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=10, startTime=1681703627111; duration=0sec 2023-04-17 03:53:47,622 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:47,623 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into ec9f51ce81ea4ad19bc57634a90f8058(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:47,623 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:47,623 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:47,623 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=10, startTime=1681703627112; duration=0sec 2023-04-17 03:53:47,623 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:47,626 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 70612 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:47,626 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:47,626 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:47,626 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/108e129382f54eafad1d34c0546833ce, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=69.0 K 2023-04-17 03:53:47,627 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 108e129382f54eafad1d34c0546833ce, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681703622335 2023-04-17 03:53:47,627 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1d4bd20858f544c893969c8cd9dd56c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1681703623137 2023-04-17 03:53:47,628 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 29280b30901c4db9a3be2f579904337c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1681703623786 2023-04-17 03:53:47,628 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2beb89fa786f4b9cbe8413dc544b208a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703624247 2023-04-17 03:53:47,629 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e9dcd7be1e764bb4988f0e9675951751, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1681703624885 2023-04-17 03:53:47,629 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2dc665b86a934a6bbd3ad1aadd8f981e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:47,644 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:47,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703687767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:47,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:47,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703687771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,027 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/e169b2bb6d584fe989280c055726f533 2023-04-17 03:53:48,052 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fb06e83f3ec14720879ea56e0916df70 2023-04-17 03:53:48,063 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/520d4c18a57c481191517e152c091006 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/520d4c18a57c481191517e152c091006 2023-04-17 03:53:48,063 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/71c1eb6d91244f06a33bd27340c7db4c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c 2023-04-17 03:53:48,072 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:53:48,074 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/e169b2bb6d584fe989280c055726f533 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533 2023-04-17 03:53:48,074 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 520d4c18a57c481191517e152c091006(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:48,076 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,076 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=10, startTime=1681703627112; duration=0sec 2023-04-17 03:53:48,076 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:48,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703688074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703688077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,082 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:53:48,083 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fb06e83f3ec14720879ea56e0916df70 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70 2023-04-17 03:53:48,089 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:53:48,090 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for b6d9193f990a9f1499ce889bc6211eef in 945ms, sequenceid=375, compaction requested=false 2023-04-17 03:53:48,090 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,090 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,090 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c615132[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,099 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,099 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:48,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,104 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,105 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,109 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:48,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,124 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,124 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:48,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:48,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:48,229 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:48,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:48,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:48,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,231 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:48,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:48,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:48,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:48,233 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:48,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,266 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/3003d05c510f4880a944ad14384799cc 2023-04-17 03:53:48,290 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/1cab156be08a4db7add81cb60c3d9caa 2023-04-17 03:53:48,308 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=449 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/e9c88416734a41bebde13c060f9f84a1 2023-04-17 03:53:48,315 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/3003d05c510f4880a944ad14384799cc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc 2023-04-17 03:53:48,325 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc, entries=150, sequenceid=449, filesize=11.9 K 2023-04-17 03:53:48,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/1cab156be08a4db7add81cb60c3d9caa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa 2023-04-17 03:53:48,332 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa, entries=150, sequenceid=449, filesize=11.9 K 2023-04-17 03:53:48,333 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/e9c88416734a41bebde13c060f9f84a1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1 2023-04-17 03:53:48,339 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1, entries=150, sequenceid=449, filesize=11.9 K 2023-04-17 03:53:48,341 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.72 KB/423648, currentSize=0 B/0 for b6d9193f990a9f1499ce889bc6211eef in 108ms, sequenceid=449, compaction requested=true 2023-04-17 03:53:48,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa934a6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,354 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,357 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,357 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,357 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,357 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,358 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,364 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,365 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:48,365 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,365 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,365 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,465 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:48,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:48,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:48,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:48,470 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:48,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:48,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:48,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:48,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:48,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,480 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49369d4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,482 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,482 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,515 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,515 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,514 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:48,593 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:48,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:48,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/c8510ee507ac46e5a8e99e2522a27238 2023-04-17 03:53:48,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:48,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:48,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:48,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:48,631 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:48,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:48,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:48,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:48,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703688632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703628619, max timestamp=1681703628634Num uniques 859; 2023-04-17 03:53:48,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703628619, max timestamp=1681703628634Num uniques 859; 2023-04-17 03:53:48,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703628619, max timestamp=1681703628634Num uniques 859; 2023-04-17 03:53:48,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:48,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:48,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:53:48,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:48,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:48,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/5b700d5df85a48cab7aa44bc80ddd80c 2023-04-17 03:53:48,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763c55f6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,648 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,648 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:48,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,654 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,654 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,660 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,665 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,666 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,666 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:48,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/9f2b632ae2c544daa140991e5d7215f8 2023-04-17 03:53:48,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/c8510ee507ac46e5a8e99e2522a27238 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238 2023-04-17 03:53:48,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238, entries=200, sequenceid=461, filesize=14.3 K 2023-04-17 03:53:48,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/5b700d5df85a48cab7aa44bc80ddd80c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c 2023-04-17 03:53:48,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c, entries=150, sequenceid=461, filesize=11.9 K 2023-04-17 03:53:48,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/9f2b632ae2c544daa140991e5d7215f8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8 2023-04-17 03:53:48,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8, entries=150, sequenceid=461, filesize=11.9 K 2023-04-17 03:53:48,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=100.63 KB/103050 for b6d9193f990a9f1499ce889bc6211eef in 106ms, sequenceid=461, compaction requested=true 2023-04-17 03:53:48,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:48,700 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:48,700 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:48,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:48,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:48,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:48,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=107.34 KB heapSize=305.30 KB 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:48,701 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:53:48,702 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:48,702 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51628 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:48,702 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:48,702 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:48,702 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,702 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,702 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ec9f51ce81ea4ad19bc57634a90f8058, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=48.0 K 2023-04-17 03:53:48,702 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/16eeca9cb729447ab932c7b90eca0793, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=50.4 K 2023-04-17 03:53:48,702 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ec9f51ce81ea4ad19bc57634a90f8058, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:48,703 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 16eeca9cb729447ab932c7b90eca0793, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:48,703 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 71c1eb6d91244f06a33bd27340c7db4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681703626564 2023-04-17 03:53:48,703 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e169b2bb6d584fe989280c055726f533, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681703626564 2023-04-17 03:53:48,704 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 3003d05c510f4880a944ad14384799cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681703627432 2023-04-17 03:53:48,705 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1cab156be08a4db7add81cb60c3d9caa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681703627432 2023-04-17 03:53:48,705 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c8510ee507ac46e5a8e99e2522a27238, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:48,705 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5b700d5df85a48cab7aa44bc80ddd80c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:48,721 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:48,723 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#53 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:48,744 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/39d0a9eff32745efa4d6ce4943594898 2023-04-17 03:53:48,755 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/b7c48a0e330949bbb347f93f679d7b3a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b7c48a0e330949bbb347f93f679d7b3a 2023-04-17 03:53:48,761 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/84da3de2376d4364b68bbad1e7c93117 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/84da3de2376d4364b68bbad1e7c93117 2023-04-17 03:53:48,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703688756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703688757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,766 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:48,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:48,771 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into 84da3de2376d4364b68bbad1e7c93117(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:48,771 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,771 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=12, startTime=1681703628699; duration=0sec 2023-04-17 03:53:48,771 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:48,771 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:48,774 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49188 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:48,774 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:48,774 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,774 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/520d4c18a57c481191517e152c091006, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=48.0 K 2023-04-17 03:53:48,775 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into b7c48a0e330949bbb347f93f679d7b3a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:48,775 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:48,775 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=12, startTime=1681703628700; duration=0sec 2023-04-17 03:53:48,775 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:48,776 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:48,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:48,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:48,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:48,780 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 520d4c18a57c481191517e152c091006, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703626193 2023-04-17 03:53:48,781 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting fb06e83f3ec14720879ea56e0916df70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681703626564 2023-04-17 03:53:48,782 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e9c88416734a41bebde13c060f9f84a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=449, earliestPutTs=1681703627432 2023-04-17 03:53:48,782 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9f2b632ae2c544daa140991e5d7215f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:48,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:48,793 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,793 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:48,793 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:48,793 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,793 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,793 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,793 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,799 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,799 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,801 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,807 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d272c2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,808 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,808 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,811 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,811 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,812 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,812 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,818 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,818 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,831 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,832 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,832 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703688864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703688864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,931 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:48,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:48,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:48,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:48,937 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:48,937 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,937 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:48,937 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:48,937 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,937 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,939 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:48,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:48,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:48,941 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:48,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:48,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:48,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:48,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:48,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,944 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:48,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@634d747a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:48,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:48,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:48,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:48,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:48,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:48,952 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:48,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:48,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:48,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:48,953 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:48,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:48,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:48,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:48,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:48,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:48,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:48,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:48,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:48,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:48,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:48,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:48,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:48,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:48,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:48,972 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:48,972 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703689066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703689066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,072 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,076 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,081 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,081 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,082 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,082 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:49,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:49,083 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:49,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:49,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0f7c0e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:49,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:49,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:49,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:49,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:49,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:49,092 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:49,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:49,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:49,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:49,092 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:49,093 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,093 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,093 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,099 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:49,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:49,100 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:49,100 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,200 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,205 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/c2c4715741a649c285f7072782bda97b 2023-04-17 03:53:49,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,210 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,214 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,216 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,216 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,216 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,216 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:49,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6808059c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:49,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:49,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:49,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:49,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:49,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:49,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:49,231 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:49,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:49,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:49,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:49,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:49,237 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,237 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,237 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,242 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/528b210cfc1640508715de8221e78e4e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/528b210cfc1640508715de8221e78e4e 2023-04-17 03:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,243 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,247 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:49,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:49,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,251 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:49,251 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,253 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 528b210cfc1640508715de8221e78e4e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:49,254 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:49,254 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=12, startTime=1681703628700; duration=0sec 2023-04-17 03:53:49,254 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:49,351 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,356 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,359 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,359 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:49,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:49,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@baf3e3d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:49,367 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:49,367 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,369 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:49,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:49,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703689371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703689371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:49,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:49,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:49,380 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:49,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:49,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:49,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:49,380 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:49,382 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,382 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,382 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,398 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:49,398 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:49,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,498 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,503 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,504 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,506 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,507 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,507 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,507 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:49,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:49,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c97596[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:49,513 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:49,513 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:49,514 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:49,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:49,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:49,519 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:49,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:49,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:49,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:49,519 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,522 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,526 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:49,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:49,529 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,529 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:49,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,633 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,635 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,636 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,636 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,638 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,638 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,638 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,638 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:49,640 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42186ed[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:49,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:49,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:49,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:49,651 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:49,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:49,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:49,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:49,651 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:49,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,653 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,653 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:49,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fbb73fb5c0f24faabe3f528025b39746 2023-04-17 03:53:49,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:49,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:49,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/39d0a9eff32745efa4d6ce4943594898 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898 2023-04-17 03:53:49,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:49,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:49,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:49,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:49,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:49,665 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:49,665 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:49,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:49,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 03:53:49,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/c2c4715741a649c285f7072782bda97b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b 2023-04-17 03:53:49,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 03:53:49,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fbb73fb5c0f24faabe3f528025b39746 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746 2023-04-17 03:53:49,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746, entries=150, sequenceid=494, filesize=11.9 K 2023-04-17 03:53:49,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~305.25 KB/312576, currentSize=80.51 KB/82440 for b6d9193f990a9f1499ce889bc6211eef in 998ms, sequenceid=494, compaction requested=false 2023-04-17 03:53:49,699 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:49,765 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:49,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:49,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:49,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:49,773 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:49,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:49,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:49,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:49,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:49,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:49,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:49,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:49,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:49,779 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:49,779 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:49,779 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:49,780 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:49,811 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/2de6d7a5a8874454ab2d06476c56d86d 2023-04-17 03:53:49,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:49,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:49,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703689874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703689875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703689875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703689878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703689878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703689978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703689980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:49,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:49,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703689983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703690183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703690184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703690186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,239 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc 2023-04-17 03:53:50,264 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/c7d55565a3004752ba5b4cd709534a03 2023-04-17 03:53:50,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/2de6d7a5a8874454ab2d06476c56d86d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d 2023-04-17 03:53:50,275 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d, entries=150, sequenceid=512, filesize=11.9 K 2023-04-17 03:53:50,276 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc 2023-04-17 03:53:50,281 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc, entries=150, sequenceid=512, filesize=11.9 K 2023-04-17 03:53:50,283 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/c7d55565a3004752ba5b4cd709534a03 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03 2023-04-17 03:53:50,289 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03, entries=150, sequenceid=512, filesize=11.9 K 2023-04-17 03:53:50,290 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for b6d9193f990a9f1499ce889bc6211eef in 511ms, sequenceid=512, compaction requested=true 2023-04-17 03:53:50,290 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:50,290 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:50,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cad5710[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:50,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:50,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,295 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:50,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:50,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:50,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:50,298 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:50,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:50,303 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:50,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:50,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:50,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:50,303 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:50,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:50,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:50,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:50,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,308 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:50,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:50,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,312 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:50,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:50,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:50,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:50,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:50,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:50,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:50,412 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:50,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:50,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:50,416 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:50,418 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:50,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:50,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:50,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:50,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:50,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:50,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:50,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:50,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:50,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:50,421 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 03:53:50,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:50,422 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:50,422 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:50,422 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:50,422 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:50,422 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:50,435 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/34b77d0c99d14cf1b4d9e5b7187636d9 2023-04-17 03:53:50,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:50,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:50,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703690504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703690505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703690507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703690608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703690609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703690614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703690815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703690819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703690819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,857 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/3f20c132dbe54cac9d3fcc07ab21e378 2023-04-17 03:53:50,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703690893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:50,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:50,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703690895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703691120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703691122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703691123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,281 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1fddf3764e8545d6af118d0736b244e0 2023-04-17 03:53:51,291 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/34b77d0c99d14cf1b4d9e5b7187636d9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9 2023-04-17 03:53:51,304 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9, entries=150, sequenceid=530, filesize=11.9 K 2023-04-17 03:53:51,305 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/3f20c132dbe54cac9d3fcc07ab21e378 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378 2023-04-17 03:53:51,313 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378, entries=150, sequenceid=530, filesize=11.9 K 2023-04-17 03:53:51,314 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1fddf3764e8545d6af118d0736b244e0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0 2023-04-17 03:53:51,324 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0, entries=150, sequenceid=530, filesize=11.9 K 2023-04-17 03:53:51,325 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b6d9193f990a9f1499ce889bc6211eef in 904ms, sequenceid=530, compaction requested=true 2023-04-17 03:53:51,325 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:51,325 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,325 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:51,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:51,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e81da6e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:51,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:51,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:51,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:51,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:51,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:51,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:51,336 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:51,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:51,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:51,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:51,336 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:51,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,338 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:51,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:51,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:51,342 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:51,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:51,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:51,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:51,342 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:51,342 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:51,442 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:51,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:51,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:51,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:51,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:51,447 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:51,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:51,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:51,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:51,448 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,448 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:51,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:51,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:51,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:51,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:51,453 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:51,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,464 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/687c38f21f2143a8ab1df2407a591bae 2023-04-17 03:53:51,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:51,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703691647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703691648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703691649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703691753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703691753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703691753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,879 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/81e8a7c41f7e4ffd94a58ca69297eb6e 2023-04-17 03:53:51,893 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/f83100b5b41d45d1b3722bb4188a9c40 2023-04-17 03:53:51,898 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/687c38f21f2143a8ab1df2407a591bae as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae 2023-04-17 03:53:51,904 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae, entries=150, sequenceid=544, filesize=11.9 K 2023-04-17 03:53:51,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/81e8a7c41f7e4ffd94a58ca69297eb6e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e 2023-04-17 03:53:51,910 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e, entries=150, sequenceid=544, filesize=11.9 K 2023-04-17 03:53:51,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/f83100b5b41d45d1b3722bb4188a9c40 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40 2023-04-17 03:53:51,916 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40, entries=150, sequenceid=544, filesize=11.9 K 2023-04-17 03:53:51,917 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b6d9193f990a9f1499ce889bc6211eef in 465ms, sequenceid=544, compaction requested=true 2023-04-17 03:53:51,917 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:51,917 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:51,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:51,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@162c71ed[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:51,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:51,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,922 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:51,922 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:51,923 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:51,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:51,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:51,924 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:51,926 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:51,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:51,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:51,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:51,927 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:51,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:51,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:51,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,932 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:51,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:51,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:51,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:51,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:51,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:51,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:51,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:51,939 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:51,939 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:51,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:51,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:51,961 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:53:51,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:51,962 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,962 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:51,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:51,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703691977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703691981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:51,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703691981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:51,992 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/95df9a63a9b948468f7b16afa523ef49 2023-04-17 03:53:52,039 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,043 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,044 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,045 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,048 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,048 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,048 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,048 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,050 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb29b2b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,056 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,059 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,059 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,059 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,059 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,059 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,072 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,072 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703692082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703692088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703692088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,173 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,178 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,186 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e15a3a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,196 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,197 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,197 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,204 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,204 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,204 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,204 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,204 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,207 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,207 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,207 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,218 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,218 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703692286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703692291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703692291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,318 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,322 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,324 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,356 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e3a9e2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,365 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,368 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,372 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,373 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,375 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,375 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,375 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,376 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/6f92fece2c3f4b328a7041b54afcc6d6 2023-04-17 03:53:52,485 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,490 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,493 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,496 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,499 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,499 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7db5120a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,505 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,515 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,515 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,518 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,522 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,525 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,527 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,527 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703692590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703692595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703692599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,627 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,631 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,636 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,638 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,639 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7754a554[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,646 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,646 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,650 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,652 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,653 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:52,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,758 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,760 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,764 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,767 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2536110e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,772 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,775 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,775 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,775 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,775 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,775 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,786 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,786 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,786 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/80ad89b452ca4210bff1e74cd6ce0183 2023-04-17 03:53:52,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/95df9a63a9b948468f7b16afa523ef49 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49 2023-04-17 03:53:52,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49, entries=200, sequenceid=564, filesize=14.3 K 2023-04-17 03:53:52,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/6f92fece2c3f4b328a7041b54afcc6d6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6 2023-04-17 03:53:52,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6, entries=100, sequenceid=564, filesize=9.5 K 2023-04-17 03:53:52,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/80ad89b452ca4210bff1e74cd6ce0183 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183 2023-04-17 03:53:52,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183, entries=100, sequenceid=564, filesize=9.5 K 2023-04-17 03:53:52,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b6d9193f990a9f1499ce889bc6211eef in 895ms, sequenceid=564, compaction requested=true 2023-04-17 03:53:52,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:52,856 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:52,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:52,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:52,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:53:52,857 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:52,858 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76132 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:52,858 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:52,859 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,859 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/84da3de2376d4364b68bbad1e7c93117, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=74.3 K 2023-04-17 03:53:52,859 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 84da3de2376d4364b68bbad1e7c93117, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:52,860 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 39d0a9eff32745efa4d6ce4943594898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681703628661 2023-04-17 03:53:52,860 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71252 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:52,861 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:52,861 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,861 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b7c48a0e330949bbb347f93f679d7b3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=69.6 K 2023-04-17 03:53:52,861 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2de6d7a5a8874454ab2d06476c56d86d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703628741 2023-04-17 03:53:52,861 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b7c48a0e330949bbb347f93f679d7b3a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:52,862 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 34b77d0c99d14cf1b4d9e5b7187636d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681703629869 2023-04-17 03:53:52,862 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c2c4715741a649c285f7072782bda97b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681703628661 2023-04-17 03:53:52,862 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 687c38f21f2143a8ab1df2407a591bae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681703630501 2023-04-17 03:53:52,863 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d4b944a34c0b4e7a8dfcbf9a9c018cbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703628741 2023-04-17 03:53:52,863 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 95df9a63a9b948468f7b16afa523ef49, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703631647 2023-04-17 03:53:52,863 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3f20c132dbe54cac9d3fcc07ab21e378, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681703629869 2023-04-17 03:53:52,864 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 81e8a7c41f7e4ffd94a58ca69297eb6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681703630501 2023-04-17 03:53:52,865 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 6f92fece2c3f4b328a7041b54afcc6d6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703631648 2023-04-17 03:53:52,883 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#69 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:52,883 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:52,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:52,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:52,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:52,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:52,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:52,894 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:52,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:52,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:52,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:52,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:52,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:52,911 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:53:52,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:52,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:52,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:52,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:52,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:52,912 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:52,915 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@308f00f3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:52,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:52,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:52,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:52,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:52,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/b43c7f5bfe944047b42a88e1340864e5 2023-04-17 03:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:52,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:52,930 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:52,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:52,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:52,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:52,930 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:52,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:52,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:52,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:52,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:52,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:52,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:52,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:52,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:52,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:52,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:52,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:52,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:52,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703692955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:52,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:52,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703692957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,046 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,047 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,049 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,052 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,052 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ff61aa[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,058 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703693058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,060 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:53,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,064 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,064 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703693064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,073 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,073 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703693095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703693099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703693104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,173 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,178 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,179 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,189 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37531644[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,193 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,193 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,204 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,208 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,208 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,211 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,216 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703693261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703693269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,304 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/1b9027619dda4cb9abb79baf86d7020f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1b9027619dda4cb9abb79baf86d7020f 2023-04-17 03:53:53,312 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into 1b9027619dda4cb9abb79baf86d7020f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:53,312 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:53,312 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=10, startTime=1681703632856; duration=0sec 2023-04-17 03:53:53,312 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:53,313 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:53:53,316 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71252 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:53:53,316 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,316 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:53,316 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,316 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/528b210cfc1640508715de8221e78e4e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=69.6 K 2023-04-17 03:53:53,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,317 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 528b210cfc1640508715de8221e78e4e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703628589 2023-04-17 03:53:53,317 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fbb73fb5c0f24faabe3f528025b39746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1681703628661 2023-04-17 03:53:53,318 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c7d55565a3004752ba5b4cd709534a03, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703628741 2023-04-17 03:53:53,319 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1fddf3764e8545d6af118d0736b244e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1681703629869 2023-04-17 03:53:53,319 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f83100b5b41d45d1b3722bb4188a9c40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1681703630501 2023-04-17 03:53:53,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,320 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 80ad89b452ca4210bff1e74cd6ce0183, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703631648 2023-04-17 03:53:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,328 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,331 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/a7b825bb77314bd19e2c4f7f0d029ca4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/a7b825bb77314bd19e2c4f7f0d029ca4 2023-04-17 03:53:53,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,336 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,346 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,349 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/4f8cc19f4be94a79b483e044b765ab5b 2023-04-17 03:53:53,351 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into a7b825bb77314bd19e2c4f7f0d029ca4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:53,351 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:53,351 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=10, startTime=1681703632856; duration=0sec 2023-04-17 03:53:53,351 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:53,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2278dfc1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,359 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,359 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,359 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:53,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,364 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,364 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,369 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,369 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,369 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,370 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,379 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,379 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:53,380 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#74 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:53,432 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/fd69d2edc23547529e58b743b6f84153 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fd69d2edc23547529e58b743b6f84153 2023-04-17 03:53:53,444 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into fd69d2edc23547529e58b743b6f84153(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:53,444 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:53,445 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=10, startTime=1681703632856; duration=0sec 2023-04-17 03:53:53,445 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:53,479 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,483 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,486 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,488 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,488 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,488 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,488 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,489 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ab8cbe5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,495 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,495 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,497 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,497 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,497 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,497 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,510 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703693563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703693575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,610 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,616 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,618 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,624 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,627 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d40529[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,636 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,640 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,640 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,665 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,666 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,666 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,766 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,770 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,774 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,775 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,775 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,775 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:53,775 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:53,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1accc1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:53,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:53,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,781 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:53,781 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:53,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:53,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:53,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:53,786 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:53,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:53,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:53,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:53,786 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:53,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,788 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:53,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,788 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:53,790 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:53,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:53,792 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:53,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:53,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:53,793 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:53,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/65a3d103da6b452ba4ba50fe4a4f8f1b 2023-04-17 03:53:53,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/b43c7f5bfe944047b42a88e1340864e5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5 2023-04-17 03:53:53,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5, entries=150, sequenceid=578, filesize=11.9 K 2023-04-17 03:53:53,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/4f8cc19f4be94a79b483e044b765ab5b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b 2023-04-17 03:53:53,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b, entries=150, sequenceid=578, filesize=11.9 K 2023-04-17 03:53:53,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/65a3d103da6b452ba4ba50fe4a4f8f1b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b 2023-04-17 03:53:53,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b, entries=150, sequenceid=578, filesize=11.9 K 2023-04-17 03:53:53,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b6d9193f990a9f1499ce889bc6211eef in 930ms, sequenceid=578, compaction requested=false 2023-04-17 03:53:53,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:53,893 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:53,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:53,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:53,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:53,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:53,897 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:53,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:53,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:53,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,900 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:53,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:53,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:53,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:53,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:53,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:53,901 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:53,902 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:53,902 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:53,902 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:53,929 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/5107efafb14b49c899f3d0a6868bd94e 2023-04-17 03:53:53,948 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/824ad54e898346b49e3e262dc359e192 2023-04-17 03:53:54,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:54,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:54,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703694098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703694107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703694109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,359 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/a23dcce9c56c418695d0c1dbe1ff64ca 2023-04-17 03:53:54,365 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/5107efafb14b49c899f3d0a6868bd94e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e 2023-04-17 03:53:54,370 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e, entries=150, sequenceid=599, filesize=11.9 K 2023-04-17 03:53:54,371 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/824ad54e898346b49e3e262dc359e192 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192 2023-04-17 03:53:54,380 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192, entries=150, sequenceid=599, filesize=11.9 K 2023-04-17 03:53:54,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/a23dcce9c56c418695d0c1dbe1ff64ca as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca 2023-04-17 03:53:54,387 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca, entries=150, sequenceid=599, filesize=11.9 K 2023-04-17 03:53:54,387 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b6d9193f990a9f1499ce889bc6211eef in 485ms, sequenceid=599, compaction requested=true 2023-04-17 03:53:54,388 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:54,388 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:54,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7886b4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:54,401 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:54,401 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:54,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 03:53:54,402 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:54,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:54,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:54,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:54,408 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:54,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:54,414 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:54,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:54,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:54,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:54,414 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:54,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:54,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:54,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:54,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:54,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,423 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:54,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,424 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:54,424 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:54,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/71fac279b6694607a7ef81291d073feb 2023-04-17 03:53:54,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/ea04fd27d10647c884c6c086c5c5fa5e 2023-04-17 03:53:54,524 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:54,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:54,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:54,536 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:54,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:54,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:54,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:54,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:54,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:54,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:54,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:54,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:54,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:54,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:54,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:54,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/27da552d0fb140f2807d97d84ef87366 2023-04-17 03:53:54,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:54,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:54,561 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:54,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:54,562 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,565 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/71fac279b6694607a7ef81291d073feb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb 2023-04-17 03:53:54,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:54,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1333855d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:54,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:54,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,571 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:54,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:54,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:54,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:54,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb, entries=150, sequenceid=615, filesize=11.9 K 2023-04-17 03:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/ea04fd27d10647c884c6c086c5c5fa5e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e 2023-04-17 03:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:54,584 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:54,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:54,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:54,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:54,584 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:54,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:54,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e, entries=150, sequenceid=615, filesize=11.9 K 2023-04-17 03:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:54,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/27da552d0fb140f2807d97d84ef87366 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366 2023-04-17 03:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:54,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:54,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:54,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:54,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:54,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,594 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:54,594 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:54,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:54,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:54,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366, entries=150, sequenceid=615, filesize=11.9 K 2023-04-17 03:53:54,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b6d9193f990a9f1499ce889bc6211eef in 196ms, sequenceid=615, compaction requested=true 2023-04-17 03:53:54,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:54,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:54,598 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:54,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:54,598 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:54,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:54,599 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:54,600 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:54,600 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:54,600 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,600 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:54,600 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/a7b825bb77314bd19e2c4f7f0d029ca4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=48.4 K 2023-04-17 03:53:54,600 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,600 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1b9027619dda4cb9abb79baf86d7020f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=48.4 K 2023-04-17 03:53:54,600 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a7b825bb77314bd19e2c4f7f0d029ca4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703630501 2023-04-17 03:53:54,601 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1b9027619dda4cb9abb79baf86d7020f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703630501 2023-04-17 03:53:54,601 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b43c7f5bfe944047b42a88e1340864e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681703631978 2023-04-17 03:53:54,602 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4f8cc19f4be94a79b483e044b765ab5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681703631978 2023-04-17 03:53:54,602 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5107efafb14b49c899f3d0a6868bd94e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681703632937 2023-04-17 03:53:54,602 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 824ad54e898346b49e3e262dc359e192, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681703632937 2023-04-17 03:53:54,603 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 71fac279b6694607a7ef81291d073feb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:54,604 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ea04fd27d10647c884c6c086c5c5fa5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:54,617 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:54,624 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:54,649 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/99680d4914d74120a99c2a755c45537e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99680d4914d74120a99c2a755c45537e 2023-04-17 03:53:54,653 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/e121ca6bcac74a7092fa659a67b722f0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/e121ca6bcac74a7092fa659a67b722f0 2023-04-17 03:53:54,658 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into 99680d4914d74120a99c2a755c45537e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:54,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:54,658 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=12, startTime=1681703634598; duration=0sec 2023-04-17 03:53:54,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:54,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:53:54,662 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:53:54,662 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:54,664 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,664 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fd69d2edc23547529e58b743b6f84153, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=48.4 K 2023-04-17 03:53:54,665 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fd69d2edc23547529e58b743b6f84153, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1681703630501 2023-04-17 03:53:54,666 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 65a3d103da6b452ba4ba50fe4a4f8f1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1681703631978 2023-04-17 03:53:54,666 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting a23dcce9c56c418695d0c1dbe1ff64ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=599, earliestPutTs=1681703632937 2023-04-17 03:53:54,666 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into e121ca6bcac74a7092fa659a67b722f0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:54,666 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:54,667 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=12, startTime=1681703634598; duration=0sec 2023-04-17 03:53:54,667 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 27da552d0fb140f2807d97d84ef87366, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:54,667 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:54,686 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#83 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:54,695 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:54,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:54,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:54,704 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:54,704 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:54,704 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:54,704 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:54,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:54,707 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,708 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:54,708 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:54,708 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1c1dd11996c842ffa26a9bbcc3302175 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1c1dd11996c842ffa26a9bbcc3302175 2023-04-17 03:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:54,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:54,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:54,712 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:54,712 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:54,712 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 03:53:54,712 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:54,715 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 1c1dd11996c842ffa26a9bbcc3302175(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:54,715 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:54,715 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=12, startTime=1681703634598; duration=0sec 2023-04-17 03:53:54,716 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:54,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:54,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:54,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703694881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:54,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703694884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703695084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703695086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,151 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/6d2a7c95f66c450a8a2856a822883e74 2023-04-17 03:53:55,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703695389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703695389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,575 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/beb14b07ec0e47d69e544da12b08ace4 2023-04-17 03:53:55,612 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/5416738959a745ec89ba33a3f7b509ee 2023-04-17 03:53:55,619 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/6d2a7c95f66c450a8a2856a822883e74 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74 2023-04-17 03:53:55,626 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74, entries=150, sequenceid=633, filesize=11.9 K 2023-04-17 03:53:55,627 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/beb14b07ec0e47d69e544da12b08ace4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4 2023-04-17 03:53:55,636 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4, entries=150, sequenceid=633, filesize=11.9 K 2023-04-17 03:53:55,638 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/5416738959a745ec89ba33a3f7b509ee as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee 2023-04-17 03:53:55,649 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee, entries=150, sequenceid=633, filesize=11.9 K 2023-04-17 03:53:55,650 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for b6d9193f990a9f1499ce889bc6211eef in 938ms, sequenceid=633, compaction requested=false 2023-04-17 03:53:55,650 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:55,650 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:55,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:55,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:55,655 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:55,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@479a275f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:55,655 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:55,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:55,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:55,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:55,662 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:55,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:55,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:55,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:55,662 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:55,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,665 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:55,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:55,665 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,667 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:55,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:55,673 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:55,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:55,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:55,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:55,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:55,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:55,774 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:55,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:55,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:55,777 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:55,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:55,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:55,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:55,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:55,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:55,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:55,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:55,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:55,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:55,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:55,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:55,782 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:55,782 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:55,782 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:55,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:55,800 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/943c54f1c4a3408aadb84e438a627cd8 2023-04-17 03:53:55,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:55,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:55,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703695908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:55,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:55,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703695909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703696014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56016 deadline: 1681703696113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56064 deadline: 1681703696113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:56052 deadline: 1681703696113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,117 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:56,119 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:56,120 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:53:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,217 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/fa6def4fd4fc479888869276bd627cca 2023-04-17 03:53:56,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703696220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,240 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1768a4310d4c4153995928e02df95b2a 2023-04-17 03:53:56,247 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/943c54f1c4a3408aadb84e438a627cd8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8 2023-04-17 03:53:56,252 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8, entries=150, sequenceid=651, filesize=11.9 K 2023-04-17 03:53:56,253 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/fa6def4fd4fc479888869276bd627cca as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca 2023-04-17 03:53:56,258 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca, entries=150, sequenceid=651, filesize=11.9 K 2023-04-17 03:53:56,259 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1768a4310d4c4153995928e02df95b2a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a 2023-04-17 03:53:56,263 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a, entries=150, sequenceid=651, filesize=11.9 K 2023-04-17 03:53:56,264 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for b6d9193f990a9f1499ce889bc6211eef in 482ms, sequenceid=651, compaction requested=true 2023-04-17 03:53:56,264 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:56,264 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:56,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:56,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:56,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:56,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:56,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aeb87a9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:56,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:56,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,269 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:56,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:56,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:56,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:56,273 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:56,273 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:56,273 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:56,273 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:56,273 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:56,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:56,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:56,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:56,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:56,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:56,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:56,381 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:56,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:56,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:56,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:56,384 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,386 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,386 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,386 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:56,386 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:56,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:56,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:56,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:56,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:56,388 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:56,388 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:56,389 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:56,389 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,398 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/107a2210ebf347bc8c02bf888e1190c5 2023-04-17 03:53:56,414 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/9cd1595063c6400eb7da0cbda986f346 2023-04-17 03:53:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:56,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:56,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703696545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703696648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,826 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/433b75b491a14e85b7766177850ac583 2023-04-17 03:53:56,832 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/107a2210ebf347bc8c02bf888e1190c5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5 2023-04-17 03:53:56,837 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5, entries=150, sequenceid=666, filesize=11.9 K 2023-04-17 03:53:56,838 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/9cd1595063c6400eb7da0cbda986f346 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346 2023-04-17 03:53:56,843 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346, entries=150, sequenceid=666, filesize=11.9 K 2023-04-17 03:53:56,844 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/433b75b491a14e85b7766177850ac583 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583 2023-04-17 03:53:56,848 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583, entries=150, sequenceid=666, filesize=11.9 K 2023-04-17 03:53:56,849 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b6d9193f990a9f1499ce889bc6211eef in 461ms, sequenceid=666, compaction requested=true 2023-04-17 03:53:56,849 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:56,849 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:56,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:56,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,851 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315df566[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:56,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:56,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:56,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-17 03:53:56,857 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:56,857 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:56,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:56,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,858 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:56,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:56,863 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:56,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:56,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:56,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:56,863 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:56,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:56,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:56,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:56,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:56,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,870 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:56,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:56,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703696893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:56,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:56,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:56,974 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:56,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:56,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:56,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:56,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:56,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:56,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:56,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:56,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:56,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:56,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:56,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:56,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:56,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:56,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4b877d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:56,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:56,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:56,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:56,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:56,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:56,988 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:56,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:56,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:56,989 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:56,989 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:56,991 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,991 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,991 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:56,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:56,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:56,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703696994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:56,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:56,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:56,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:56,999 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:56,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:56,999 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:56,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:57,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703697002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,099 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,103 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,104 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,106 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,106 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,106 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,107 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,109 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28fdff34[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,111 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,111 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,116 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,117 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,119 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,120 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,125 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,125 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:57,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703697198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:57,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703697206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,225 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,229 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,234 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebde34d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,237 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,237 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,238 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,238 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,252 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,252 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,254 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,255 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:57,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/0301332d710b42bb94bad09fb32a591c 2023-04-17 03:53:57,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,362 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,366 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,366 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,366 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,366 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,368 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6095c739[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,373 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,374 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,377 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,377 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,383 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,383 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,383 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:57,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,488 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,494 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,494 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,494 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,495 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,496 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@287d439[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,500 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,505 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,505 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703697504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703697509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,511 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:57,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,517 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,517 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,617 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,626 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,634 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,634 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,634 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,634 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,636 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45448f02[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,639 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,639 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,646 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,647 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,650 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,661 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,661 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/9324ffd260bd4c99bf1b54081e57872a 2023-04-17 03:53:57,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/b9cae70c37594e08a15d22f09bf2bbb9 2023-04-17 03:53:57,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/0301332d710b42bb94bad09fb32a591c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c 2023-04-17 03:53:57,761 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c, entries=150, sequenceid=685, filesize=11.9 K 2023-04-17 03:53:57,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/9324ffd260bd4c99bf1b54081e57872a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a 2023-04-17 03:53:57,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,767 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a, entries=150, sequenceid=685, filesize=11.9 K 2023-04-17 03:53:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/b9cae70c37594e08a15d22f09bf2bbb9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9 2023-04-17 03:53:57,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:57,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:57,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:57,775 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,776 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b2bb6c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:57,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:57,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9, entries=150, sequenceid=685, filesize=11.9 K 2023-04-17 03:53:57,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for b6d9193f990a9f1499ce889bc6211eef in 927ms, sequenceid=685, compaction requested=true 2023-04-17 03:53:57,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:57,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:57,783 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:53:57,783 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:53:57,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:57,783 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:57,783 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:57,784 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:57,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:57,786 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,786 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:57,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:57,791 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:57,791 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:53:57,791 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:57,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:57,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:57,792 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:57,792 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/A is initiating minor compaction (all files) 2023-04-17 03:53:57,792 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/A in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,792 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/e121ca6bcac74a7092fa659a67b722f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=60.4 K 2023-04-17 03:53:57,792 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:53:57,792 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/B is initiating minor compaction (all files) 2023-04-17 03:53:57,792 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/B in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,792 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99680d4914d74120a99c2a755c45537e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=60.4 K 2023-04-17 03:53:57,793 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e121ca6bcac74a7092fa659a67b722f0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:57,793 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 99680d4914d74120a99c2a755c45537e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:57,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,793 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6d2a7c95f66c450a8a2856a822883e74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1681703634434 2023-04-17 03:53:57,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:57,794 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting beb14b07ec0e47d69e544da12b08ace4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1681703634434 2023-04-17 03:53:57,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,793 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:57,794 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 943c54f1c4a3408aadb84e438a627cd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681703634773 2023-04-17 03:53:57,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,794 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fa6def4fd4fc479888869276bd627cca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681703634773 2023-04-17 03:53:57,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,795 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 107a2210ebf347bc8c02bf888e1190c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681703635903 2023-04-17 03:53:57,795 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9cd1595063c6400eb7da0cbda986f346, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681703635903 2023-04-17 03:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:57,795 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0301332d710b42bb94bad09fb32a591c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681703636543 2023-04-17 03:53:57,795 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9324ffd260bd4c99bf1b54081e57872a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681703636543 2023-04-17 03:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:57,796 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:57,801 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,801 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,801 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:57,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:57,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:57,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,802 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:57,803 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:57,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:57,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:53:57,814 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#B#compaction#96 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:57,816 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#A#compaction#97 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:57,833 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/4ba4fef34eea44aebf1dc7859eaa8097 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4ba4fef34eea44aebf1dc7859eaa8097 2023-04-17 03:53:57,843 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/B of b6d9193f990a9f1499ce889bc6211eef into 4ba4fef34eea44aebf1dc7859eaa8097(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:57,843 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:57,843 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/B, priority=11, startTime=1681703637783; duration=0sec 2023-04-17 03:53:57,843 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:53:57,843 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:53:57,847 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:53:57,847 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): b6d9193f990a9f1499ce889bc6211eef/C is initiating minor compaction (all files) 2023-04-17 03:53:57,847 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b6d9193f990a9f1499ce889bc6211eef/C in TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,847 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1c1dd11996c842ffa26a9bbcc3302175, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp, totalSize=60.4 K 2023-04-17 03:53:57,848 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1c1dd11996c842ffa26a9bbcc3302175, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=615, earliestPutTs=1681703634080 2023-04-17 03:53:57,848 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5416738959a745ec89ba33a3f7b509ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1681703634434 2023-04-17 03:53:57,849 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1768a4310d4c4153995928e02df95b2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1681703634773 2023-04-17 03:53:57,850 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 433b75b491a14e85b7766177850ac583, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1681703635903 2023-04-17 03:53:57,850 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b9cae70c37594e08a15d22f09bf2bbb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1681703636543 2023-04-17 03:53:57,851 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/c151043eb6a84c68b9613c5fe2459eec as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c151043eb6a84c68b9613c5fe2459eec 2023-04-17 03:53:57,859 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/A of b6d9193f990a9f1499ce889bc6211eef into c151043eb6a84c68b9613c5fe2459eec(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:57,859 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:57,859 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/A, priority=11, startTime=1681703637783; duration=0sec 2023-04-17 03:53:57,859 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:57,868 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): b6d9193f990a9f1499ce889bc6211eef#C#compaction#98 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:53:57,883 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/3a125dc83e384062b65cbcece7863c02 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/3a125dc83e384062b65cbcece7863c02 2023-04-17 03:53:57,888 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b6d9193f990a9f1499ce889bc6211eef/C of b6d9193f990a9f1499ce889bc6211eef into 3a125dc83e384062b65cbcece7863c02(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:53:57,889 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:57,889 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef., storeName=b6d9193f990a9f1499ce889bc6211eef/C, priority=11, startTime=1681703637783; duration=0sec 2023-04-17 03:53:57,889 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:53:57,903 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:57,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:53:57,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:53:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:53:57,906 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:53:57,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:53:57,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:53:57,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,907 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,909 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:57,909 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:57,909 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:57,909 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:53:57,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:57,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:53:57,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:53:57,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:57,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. started... 2023-04-17 03:53:57,912 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:53:57,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:53:57,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:57,913 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:53:57,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:53:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:53:58,011 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. as already flushing 2023-04-17 03:53:58,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703698039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703698040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703698143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703698143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,327 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9c3eacef695b4b1bbc67c2987e22df9b 2023-04-17 03:53:58,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703698345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703698345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703698647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703698649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:58,740 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/99d42104bc304f59a6564c4a9560e3ea 2023-04-17 03:53:58,869 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:60620 2023-04-17 03:53:58,869 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:53:58,871 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:60620 2023-04-17 03:53:58,871 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:53:58,883 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:60620 2023-04-17 03:53:58,883 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:53:58,885 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bd6fc59 to 127.0.0.1:60620 2023-04-17 03:53:58,885 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:53:59,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:59,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56044 deadline: 1681703699149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:53:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:56024 deadline: 1681703699155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,159 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1856ef0fdba143ffb08ef47fd464181e 2023-04-17 03:53:59,165 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9c3eacef695b4b1bbc67c2987e22df9b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9c3eacef695b4b1bbc67c2987e22df9b 2023-04-17 03:53:59,168 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9c3eacef695b4b1bbc67c2987e22df9b, entries=150, sequenceid=701, filesize=11.9 K 2023-04-17 03:53:59,169 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/99d42104bc304f59a6564c4a9560e3ea as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99d42104bc304f59a6564c4a9560e3ea 2023-04-17 03:53:59,173 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99d42104bc304f59a6564c4a9560e3ea, entries=150, sequenceid=701, filesize=11.9 K 2023-04-17 03:53:59,174 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/1856ef0fdba143ffb08ef47fd464181e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1856ef0fdba143ffb08ef47fd464181e 2023-04-17 03:53:59,179 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1856ef0fdba143ffb08ef47fd464181e, entries=150, sequenceid=701, filesize=11.9 K 2023-04-17 03:53:59,180 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b6d9193f990a9f1499ce889bc6211eef in 1268ms, sequenceid=701, compaction requested=false 2023-04-17 03:53:59,180 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:53:59,180 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:53:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:53:59,184 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:53:59,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad00acc[Count = 0] remaining members to acquire global barrier 2023-04-17 03:53:59,186 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:53:59,186 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,187 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:53:59,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:53:59,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:53:59,187 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,187 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:53:59,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:53:59,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:59,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:59,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:59,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:53:59,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:53:59,192 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:53:59,192 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:53:59,192 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:53:59,192 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:53:59,192 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:53:59,194 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,194 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:53:59,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,194 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:53:59,194 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:59,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:53:59,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:53:59,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,197 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:59,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:53:59,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:53:59,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:53:59,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:53:59,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:53:59,198 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:53:59,198 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:53:59,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:53:59,298 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:53:59,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:00,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:54:00,126 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:54:00,126 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:60620 2023-04-17 03:54:00,126 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:54:00,127 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:00,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:54:00,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:00,127 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:54:00,127 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:00,138 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:60620 2023-04-17 03:54:00,139 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,151 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:60620 2023-04-17 03:54:00,151 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,158 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:60620 2023-04-17 03:54:00,158 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,163 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:60620 2023-04-17 03:54:00,163 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 198 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 246 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 58 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4016 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 3994 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1797 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5388 rows 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1780 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5337 rows 2023-04-17 03:54:00,163 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:54:00,164 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b17e1c1 to 127.0.0.1:60620 2023-04-17 03:54:00,164 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:00,169 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:54:00,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:54:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:00,183 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703640183"}]},"ts":"1681703640183"} 2023-04-17 03:54:00,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:00,185 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:54:00,186 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:54:00,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, UNASSIGN}] 2023-04-17 03:54:00,189 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, UNASSIGN 2023-04-17 03:54:00,190 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b6d9193f990a9f1499ce889bc6211eef, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:00,190 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703640190"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703640190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703640190"}]},"ts":"1681703640190"} 2023-04-17 03:54:00,192 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:00,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:00,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:54:00,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b6d9193f990a9f1499ce889bc6211eef, disabling compactions & flushes 2023-04-17 03:54:00,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:54:00,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:00,534 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/0cd21f531c0c4b1bad5d89584eecaba4 2023-04-17 03:54:00,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:00,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/54e8d4c2fb8549c1ae1ef813acfd0e7b 2023-04-17 03:54:01,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:01,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/cb701ac2ccdf430d98b827250c89e8dc 2023-04-17 03:54:01,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/0cd21f531c0c4b1bad5d89584eecaba4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0cd21f531c0c4b1bad5d89584eecaba4 2023-04-17 03:54:01,370 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0cd21f531c0c4b1bad5d89584eecaba4, entries=150, sequenceid=721, filesize=11.9 K 2023-04-17 03:54:01,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/54e8d4c2fb8549c1ae1ef813acfd0e7b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/54e8d4c2fb8549c1ae1ef813acfd0e7b 2023-04-17 03:54:01,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/54e8d4c2fb8549c1ae1ef813acfd0e7b, entries=150, sequenceid=721, filesize=11.9 K 2023-04-17 03:54:01,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/cb701ac2ccdf430d98b827250c89e8dc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/cb701ac2ccdf430d98b827250c89e8dc 2023-04-17 03:54:01,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/cb701ac2ccdf430d98b827250c89e8dc, entries=150, sequenceid=721, filesize=11.9 K 2023-04-17 03:54:01,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for b6d9193f990a9f1499ce889bc6211eef in 1256ms, sequenceid=721, compaction requested=true 2023-04-17 03:54:01,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:54:01,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:54:01,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:01,382 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. because compaction request was cancelled 2023-04-17 03:54:01,382 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. because compaction request was cancelled 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:54:01,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. after waiting 0 ms 2023-04-17 03:54:01,382 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. because compaction request was cancelled 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:54:01,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:01,382 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b6d9193f990a9f1499ce889bc6211eef 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=A 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=B 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b6d9193f990a9f1499ce889bc6211eef, store=C 2023-04-17 03:54:01,382 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:01,789 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9b6be35ff51a49e9b8a55b2b2ac68b47 2023-04-17 03:54:02,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/62ab33fc8a614865a80ffd4e24bd09fc 2023-04-17 03:54:02,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:02,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=728 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/d2be4a0aa2264ab0accbb4b5bce4a109 2023-04-17 03:54:02,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/A/9b6be35ff51a49e9b8a55b2b2ac68b47 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b6be35ff51a49e9b8a55b2b2ac68b47 2023-04-17 03:54:02,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b6be35ff51a49e9b8a55b2b2ac68b47, entries=150, sequenceid=728, filesize=11.9 K 2023-04-17 03:54:02,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/B/62ab33fc8a614865a80ffd4e24bd09fc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/62ab33fc8a614865a80ffd4e24bd09fc 2023-04-17 03:54:02,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/62ab33fc8a614865a80ffd4e24bd09fc, entries=150, sequenceid=728, filesize=11.9 K 2023-04-17 03:54:02,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/.tmp/C/d2be4a0aa2264ab0accbb4b5bce4a109 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/d2be4a0aa2264ab0accbb4b5bce4a109 2023-04-17 03:54:02,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/d2be4a0aa2264ab0accbb4b5bce4a109, entries=150, sequenceid=728, filesize=11.9 K 2023-04-17 03:54:02,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for b6d9193f990a9f1499ce889bc6211eef in 1253ms, sequenceid=728, compaction requested=true 2023-04-17 03:54:02,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af4a2d262195451c919e6ec000385326, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/8234740a35b541f2a325d6b53851aa04, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/16eeca9cb729447ab932c7b90eca0793, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/84da3de2376d4364b68bbad1e7c93117, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/a7b825bb77314bd19e2c4f7f0d029ca4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/e121ca6bcac74a7092fa659a67b722f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c] to archive 2023-04-17 03:54:02,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:02,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/43410c1d428e41caba2f5ab10837e685 2023-04-17 03:54:02,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/251bf04cb37c47d28797b78e47f09950 2023-04-17 03:54:02,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b3cfd282401f42cfaeca6201a52aaaa7 2023-04-17 03:54:02,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/dccc34921c034d0aa9d0fb0f48c023b6 2023-04-17 03:54:02,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af4a2d262195451c919e6ec000385326 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af4a2d262195451c919e6ec000385326 2023-04-17 03:54:02,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b8d0a6f1f6c4beb9edbca27b916e218 2023-04-17 03:54:02,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/290bfe7855d84458a8fa31bd4e955ebc 2023-04-17 03:54:02,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/8234740a35b541f2a325d6b53851aa04 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/8234740a35b541f2a325d6b53851aa04 2023-04-17 03:54:02,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/af6b921e082c4806b319156cb6c37e81 2023-04-17 03:54:02,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/04226128fac34f95aa9a10c2a0ff3213 2023-04-17 03:54:02,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/1f7349c58efc423d950264ed60629d6f 2023-04-17 03:54:02,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/91d6d9fca5214c54aaab457d98ffb9e8 2023-04-17 03:54:02,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/abae228356174384bb575a448b6fa08b 2023-04-17 03:54:02,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/16eeca9cb729447ab932c7b90eca0793 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/16eeca9cb729447ab932c7b90eca0793 2023-04-17 03:54:02,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71c1eb6d91244f06a33bd27340c7db4c 2023-04-17 03:54:02,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/3003d05c510f4880a944ad14384799cc 2023-04-17 03:54:02,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c8510ee507ac46e5a8e99e2522a27238 2023-04-17 03:54:02,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/84da3de2376d4364b68bbad1e7c93117 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/84da3de2376d4364b68bbad1e7c93117 2023-04-17 03:54:02,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/39d0a9eff32745efa4d6ce4943594898 2023-04-17 03:54:02,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/2de6d7a5a8874454ab2d06476c56d86d 2023-04-17 03:54:02,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/34b77d0c99d14cf1b4d9e5b7187636d9 2023-04-17 03:54:02,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/687c38f21f2143a8ab1df2407a591bae 2023-04-17 03:54:02,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/95df9a63a9b948468f7b16afa523ef49 2023-04-17 03:54:02,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/a7b825bb77314bd19e2c4f7f0d029ca4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/a7b825bb77314bd19e2c4f7f0d029ca4 2023-04-17 03:54:02,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/b43c7f5bfe944047b42a88e1340864e5 2023-04-17 03:54:02,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/5107efafb14b49c899f3d0a6868bd94e 2023-04-17 03:54:02,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/e121ca6bcac74a7092fa659a67b722f0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/e121ca6bcac74a7092fa659a67b722f0 2023-04-17 03:54:02,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/71fac279b6694607a7ef81291d073feb 2023-04-17 03:54:02,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/6d2a7c95f66c450a8a2856a822883e74 2023-04-17 03:54:02,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/943c54f1c4a3408aadb84e438a627cd8 2023-04-17 03:54:02,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/107a2210ebf347bc8c02bf888e1190c5 2023-04-17 03:54:02,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0301332d710b42bb94bad09fb32a591c 2023-04-17 03:54:02,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/21e7970f9a8646248c8809c3574eef80, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f05386bac67147629a1f18b17a91cfba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ec9f51ce81ea4ad19bc57634a90f8058, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b7c48a0e330949bbb347f93f679d7b3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1b9027619dda4cb9abb79baf86d7020f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99680d4914d74120a99c2a755c45537e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a] to archive 2023-04-17 03:54:02,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:02,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8821ab6a5434412abef56a3b3d3991fa 2023-04-17 03:54:02,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5f0b782bcc3b4877a9a50d7e98e3c65b 2023-04-17 03:54:02,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1c2a41cff28141309ad4c7dc67616180 2023-04-17 03:54:02,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/21e7970f9a8646248c8809c3574eef80 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/21e7970f9a8646248c8809c3574eef80 2023-04-17 03:54:02,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f54ceb9bd3d74e1eb651fd1be43bb3ef 2023-04-17 03:54:02,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/2bf2b93b0c6f49a3966d89a3e0e72f91 2023-04-17 03:54:02,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f05386bac67147629a1f18b17a91cfba to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/f05386bac67147629a1f18b17a91cfba 2023-04-17 03:54:02,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/da2d0ca0814f4efb9246721ec38d6cf5 2023-04-17 03:54:02,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/8989136176f742febb59058cd0691f4b 2023-04-17 03:54:02,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c6bcacf514c74572becd4b0bb37a3a80 2023-04-17 03:54:02,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/299e5e554ef043a0bdde8aad7fdf1699 2023-04-17 03:54:02,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b07ee0166cfd41638ad686cb89f9c365 2023-04-17 03:54:02,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ec9f51ce81ea4ad19bc57634a90f8058 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ec9f51ce81ea4ad19bc57634a90f8058 2023-04-17 03:54:02,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6c1eefdfeb0b4506b86969046ec82210 2023-04-17 03:54:02,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/e169b2bb6d584fe989280c055726f533 2023-04-17 03:54:02,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1cab156be08a4db7add81cb60c3d9caa 2023-04-17 03:54:02,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b7c48a0e330949bbb347f93f679d7b3a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/b7c48a0e330949bbb347f93f679d7b3a 2023-04-17 03:54:02,772 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/5b700d5df85a48cab7aa44bc80ddd80c 2023-04-17 03:54:02,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/c2c4715741a649c285f7072782bda97b 2023-04-17 03:54:02,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/d4b944a34c0b4e7a8dfcbf9a9c018cbc 2023-04-17 03:54:02,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/3f20c132dbe54cac9d3fcc07ab21e378 2023-04-17 03:54:02,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/81e8a7c41f7e4ffd94a58ca69297eb6e 2023-04-17 03:54:02,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1b9027619dda4cb9abb79baf86d7020f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/1b9027619dda4cb9abb79baf86d7020f 2023-04-17 03:54:02,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/6f92fece2c3f4b328a7041b54afcc6d6 2023-04-17 03:54:02,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4f8cc19f4be94a79b483e044b765ab5b 2023-04-17 03:54:02,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/824ad54e898346b49e3e262dc359e192 2023-04-17 03:54:02,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99680d4914d74120a99c2a755c45537e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99680d4914d74120a99c2a755c45537e 2023-04-17 03:54:02,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/ea04fd27d10647c884c6c086c5c5fa5e 2023-04-17 03:54:02,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/beb14b07ec0e47d69e544da12b08ace4 2023-04-17 03:54:02,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/fa6def4fd4fc479888869276bd627cca 2023-04-17 03:54:02,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9cd1595063c6400eb7da0cbda986f346 2023-04-17 03:54:02,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/9324ffd260bd4c99bf1b54081e57872a 2023-04-17 03:54:02,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/109748810eb141d899bd27c456e46aa9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/108e129382f54eafad1d34c0546833ce, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/520d4c18a57c481191517e152c091006, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/528b210cfc1640508715de8221e78e4e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fd69d2edc23547529e58b743b6f84153, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1c1dd11996c842ffa26a9bbcc3302175, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9] to archive 2023-04-17 03:54:02,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:02,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fc0fadbe51a34ba6a002e54a293eb574 2023-04-17 03:54:02,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9c04d32fdb7c4226869556e774bc156b 2023-04-17 03:54:02,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/0bf994dfcf05429694360acbda6ef11a 2023-04-17 03:54:02,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/109748810eb141d899bd27c456e46aa9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/109748810eb141d899bd27c456e46aa9 2023-04-17 03:54:02,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/8de6966b3ebc40469ddf5e3f280d5c76 2023-04-17 03:54:02,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/ef53a3d7281344cf8182c17c85eab2c4 2023-04-17 03:54:02,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/108e129382f54eafad1d34c0546833ce to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/108e129382f54eafad1d34c0546833ce 2023-04-17 03:54:02,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/7c5bac05ad93416dbb22eabac0a636fe 2023-04-17 03:54:02,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1d4bd20858f544c893969c8cd9dd56c6 2023-04-17 03:54:02,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/29280b30901c4db9a3be2f579904337c 2023-04-17 03:54:02,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2beb89fa786f4b9cbe8413dc544b208a 2023-04-17 03:54:02,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9dcd7be1e764bb4988f0e9675951751 2023-04-17 03:54:02,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/520d4c18a57c481191517e152c091006 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/520d4c18a57c481191517e152c091006 2023-04-17 03:54:02,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/2dc665b86a934a6bbd3ad1aadd8f981e 2023-04-17 03:54:02,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fb06e83f3ec14720879ea56e0916df70 2023-04-17 03:54:02,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/e9c88416734a41bebde13c060f9f84a1 2023-04-17 03:54:02,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/528b210cfc1640508715de8221e78e4e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/528b210cfc1640508715de8221e78e4e 2023-04-17 03:54:02,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/9f2b632ae2c544daa140991e5d7215f8 2023-04-17 03:54:02,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fbb73fb5c0f24faabe3f528025b39746 2023-04-17 03:54:02,816 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/c7d55565a3004752ba5b4cd709534a03 2023-04-17 03:54:02,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1fddf3764e8545d6af118d0736b244e0 2023-04-17 03:54:02,817 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/f83100b5b41d45d1b3722bb4188a9c40 2023-04-17 03:54:02,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fd69d2edc23547529e58b743b6f84153 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/fd69d2edc23547529e58b743b6f84153 2023-04-17 03:54:02,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/80ad89b452ca4210bff1e74cd6ce0183 2023-04-17 03:54:02,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/65a3d103da6b452ba4ba50fe4a4f8f1b 2023-04-17 03:54:02,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/a23dcce9c56c418695d0c1dbe1ff64ca 2023-04-17 03:54:02,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1c1dd11996c842ffa26a9bbcc3302175 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1c1dd11996c842ffa26a9bbcc3302175 2023-04-17 03:54:02,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/27da552d0fb140f2807d97d84ef87366 2023-04-17 03:54:02,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/5416738959a745ec89ba33a3f7b509ee 2023-04-17 03:54:02,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1768a4310d4c4153995928e02df95b2a 2023-04-17 03:54:02,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/433b75b491a14e85b7766177850ac583 2023-04-17 03:54:02,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/b9cae70c37594e08a15d22f09bf2bbb9 2023-04-17 03:54:02,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/recovered.edits/731.seqid, newMaxSeqId=731, maxSeqId=1 2023-04-17 03:54:02,833 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef. 2023-04-17 03:54:02,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b6d9193f990a9f1499ce889bc6211eef: 2023-04-17 03:54:02,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:54:02,835 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b6d9193f990a9f1499ce889bc6211eef, regionState=CLOSED 2023-04-17 03:54:02,835 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703642835"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703642835"}]},"ts":"1681703642835"} 2023-04-17 03:54:02,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-17 03:54:02,838 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure b6d9193f990a9f1499ce889bc6211eef, server=jenkins-hbase4.apache.org,36381,1681703614253 in 2.6440 sec 2023-04-17 03:54:02,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-17 03:54:02,839 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b6d9193f990a9f1499ce889bc6211eef, UNASSIGN in 2.6500 sec 2023-04-17 03:54:02,840 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703642840"}]},"ts":"1681703642840"} 2023-04-17 03:54:02,841 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:54:02,842 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:54:02,844 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6680 sec 2023-04-17 03:54:04,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-17 03:54:04,289 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-17 03:54:04,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:54:04,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,303 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 03:54:04,307 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:54:04,310 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/recovered.edits] 2023-04-17 03:54:04,313 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0cd21f531c0c4b1bad5d89584eecaba4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/0cd21f531c0c4b1bad5d89584eecaba4 2023-04-17 03:54:04,314 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b6be35ff51a49e9b8a55b2b2ac68b47 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9b6be35ff51a49e9b8a55b2b2ac68b47 2023-04-17 03:54:04,316 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9c3eacef695b4b1bbc67c2987e22df9b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/9c3eacef695b4b1bbc67c2987e22df9b 2023-04-17 03:54:04,317 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c151043eb6a84c68b9613c5fe2459eec to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/A/c151043eb6a84c68b9613c5fe2459eec 2023-04-17 03:54:04,319 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4ba4fef34eea44aebf1dc7859eaa8097 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/4ba4fef34eea44aebf1dc7859eaa8097 2023-04-17 03:54:04,321 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/54e8d4c2fb8549c1ae1ef813acfd0e7b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/54e8d4c2fb8549c1ae1ef813acfd0e7b 2023-04-17 03:54:04,322 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/62ab33fc8a614865a80ffd4e24bd09fc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/62ab33fc8a614865a80ffd4e24bd09fc 2023-04-17 03:54:04,323 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99d42104bc304f59a6564c4a9560e3ea to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/B/99d42104bc304f59a6564c4a9560e3ea 2023-04-17 03:54:04,326 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1856ef0fdba143ffb08ef47fd464181e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/1856ef0fdba143ffb08ef47fd464181e 2023-04-17 03:54:04,327 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/3a125dc83e384062b65cbcece7863c02 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/3a125dc83e384062b65cbcece7863c02 2023-04-17 03:54:04,329 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/cb701ac2ccdf430d98b827250c89e8dc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/cb701ac2ccdf430d98b827250c89e8dc 2023-04-17 03:54:04,330 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/d2be4a0aa2264ab0accbb4b5bce4a109 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/C/d2be4a0aa2264ab0accbb4b5bce4a109 2023-04-17 03:54:04,333 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/recovered.edits/731.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef/recovered.edits/731.seqid 2023-04-17 03:54:04,333 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/b6d9193f990a9f1499ce889bc6211eef 2023-04-17 03:54:04,333 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:54:04,336 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,358 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:54:04,402 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:54:04,403 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,403 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:54:04,403 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703644403"}]},"ts":"9223372036854775807"} 2023-04-17 03:54:04,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 03:54:04,406 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:54:04,406 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b6d9193f990a9f1499ce889bc6211eef, NAME => 'TestAcidGuarantees,,1681703617602.b6d9193f990a9f1499ce889bc6211eef.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:54:04,406 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:54:04,406 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703644406"}]},"ts":"9223372036854775807"} 2023-04-17 03:54:04,408 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:54:04,410 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,411 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 115 msec 2023-04-17 03:54:04,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-17 03:54:04,606 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-17 03:54:04,622 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=235 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_299733188_15 at /127.0.0.1:39216 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:36381-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x5c683274-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_299733188_15 at /127.0.0.1:39270 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c683274-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c683274-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5c683274-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=566 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=390 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=812 (was 1478) 2023-04-17 03:54:04,636 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=235, OpenFileDescriptor=566, MaxFileDescriptor=60000, SystemLoadAverage=390, ProcessCount=169, AvailableMemoryMB=811 2023-04-17 03:54:04,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:54:04,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:54:04,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:04,642 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:54:04,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-17 03:54:04,643 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:54:04,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:04,645 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:04,646 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 empty. 2023-04-17 03:54:04,646 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:04,646 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:54:04,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:04,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:05,058 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:54:05,059 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c6ba4e4e990f87093f177b6844dd61b9, NAME => 'TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:54:05,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c6ba4e4e990f87093f177b6844dd61b9, disabling compactions & flushes 2023-04-17 03:54:05,466 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. after waiting 0 ms 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,466 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,466 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:05,469 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:54:05,470 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703645470"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703645470"}]},"ts":"1681703645470"} 2023-04-17 03:54:05,472 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:54:05,473 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:54:05,473 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703645473"}]},"ts":"1681703645473"} 2023-04-17 03:54:05,474 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:54:05,478 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, ASSIGN}] 2023-04-17 03:54:05,480 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, ASSIGN 2023-04-17 03:54:05,481 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:54:05,632 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:05,633 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703645632"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703645632"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703645632"}]},"ts":"1681703645632"} 2023-04-17 03:54:05,635 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:05,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:05,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c6ba4e4e990f87093f177b6844dd61b9, NAME => 'TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:54:05,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:54:05,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,797 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,799 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A 2023-04-17 03:54:05,799 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A 2023-04-17 03:54:05,799 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:05,799 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName A 2023-04-17 03:54:05,800 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:05,800 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,801 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B 2023-04-17 03:54:05,801 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B 2023-04-17 03:54:05,801 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:05,801 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName B 2023-04-17 03:54:05,802 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:05,802 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,803 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C 2023-04-17 03:54:05,803 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C 2023-04-17 03:54:05,804 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:05,804 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName C 2023-04-17 03:54:05,804 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:05,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:54:05,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:05,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:54:05,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c6ba4e4e990f87093f177b6844dd61b9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72141345, jitterRate=0.07498981058597565}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:54:05,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:05,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., pid=18, masterSystemTime=1681703645788 2023-04-17 03:54:05,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:05,821 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:05,821 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703645821"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703645821"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703645821"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703645821"}]},"ts":"1681703645821"} 2023-04-17 03:54:05,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-17 03:54:05,825 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 in 188 msec 2023-04-17 03:54:05,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-17 03:54:05,827 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, ASSIGN in 347 msec 2023-04-17 03:54:05,827 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:54:05,828 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703645827"}]},"ts":"1681703645827"} 2023-04-17 03:54:05,829 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:54:05,831 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:54:05,833 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1920 sec 2023-04-17 03:54:06,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-17 03:54:06,749 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-17 03:54:06,750 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0186ae08 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:06,757 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@324dc814, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:06,758 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:06,762 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36210, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:06,772 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:54:06,776 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40022, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:54:06,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:54:06,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:54:06,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:07,211 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:54:07,213 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:54:07,213 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:54:07,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 03:54:07,232 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, REOPEN/MOVE}] 2023-04-17 03:54:07,233 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, REOPEN/MOVE 2023-04-17 03:54:07,234 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,234 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703647234"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703647234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703647234"}]},"ts":"1681703647234"} 2023-04-17 03:54:07,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:07,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c6ba4e4e990f87093f177b6844dd61b9, disabling compactions & flushes 2023-04-17 03:54:07,389 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. after waiting 0 ms 2023-04-17 03:54:07,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,400 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 03:54:07,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:07,401 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c6ba4e4e990f87093f177b6844dd61b9 to self. 2023-04-17 03:54:07,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,404 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=CLOSED 2023-04-17 03:54:07,404 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703647404"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703647404"}]},"ts":"1681703647404"} 2023-04-17 03:54:07,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-17 03:54:07,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 in 171 msec 2023-04-17 03:54:07,409 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=true 2023-04-17 03:54:07,559 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,559 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703647559"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703647559"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703647559"}]},"ts":"1681703647559"} 2023-04-17 03:54:07,561 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:07,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c6ba4e4e990f87093f177b6844dd61b9, NAME => 'TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:54:07,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:54:07,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,716 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,719 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,719 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A 2023-04-17 03:54:07,720 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A 2023-04-17 03:54:07,720 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:07,722 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName A 2023-04-17 03:54:07,724 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:07,725 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,726 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B 2023-04-17 03:54:07,726 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B 2023-04-17 03:54:07,726 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:07,727 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName B 2023-04-17 03:54:07,727 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:07,727 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,728 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C 2023-04-17 03:54:07,728 DEBUG [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C 2023-04-17 03:54:07,728 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:07,729 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ba4e4e990f87093f177b6844dd61b9 columnFamilyName C 2023-04-17 03:54:07,729 INFO [StoreOpener-c6ba4e4e990f87093f177b6844dd61b9-1] regionserver.HStore(310): Store=c6ba4e4e990f87093f177b6844dd61b9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:07,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,730 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:54:07,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c6ba4e4e990f87093f177b6844dd61b9; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69627351, jitterRate=0.0375283807516098}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:54:07,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:07,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., pid=23, masterSystemTime=1681703647713 2023-04-17 03:54:07,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,737 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,738 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703647737"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703647737"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703647737"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703647737"}]},"ts":"1681703647737"} 2023-04-17 03:54:07,740 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-17 03:54:07,740 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 in 178 msec 2023-04-17 03:54:07,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-17 03:54:07,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, REOPEN/MOVE in 508 msec 2023-04-17 03:54:07,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-17 03:54:07,744 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 527 msec 2023-04-17 03:54:07,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 957 msec 2023-04-17 03:54:07,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-17 03:54:07,754 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x33a47c05 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,759 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bbdba17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,760 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,764 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,765 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,768 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,769 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,773 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,774 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,777 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,777 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,781 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,782 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,785 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,785 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,790 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,790 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x092bed09 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:07,793 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42d325f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:07,799 DEBUG [hconnection-0x2972da0d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,799 DEBUG [hconnection-0x1ec56119-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,799 DEBUG [hconnection-0x1696913d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,799 DEBUG [hconnection-0x29fc2956-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,801 DEBUG [hconnection-0x42832681-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:07,801 DEBUG [hconnection-0x5bf8eed2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,802 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,807 DEBUG [hconnection-0x25adaa4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,808 DEBUG [hconnection-0x4f23b9b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,808 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,808 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,808 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,808 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,809 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,813 DEBUG [hconnection-0x151c3500-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:07,819 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36282, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,825 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:07,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:07,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:07,825 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:07,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:07,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:07,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:07,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:07,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:07,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:07,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:07,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:07,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:07,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:07,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:07,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:07,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:07,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:07,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:07,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:07,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:07,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:07,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:07,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:07,842 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:07,842 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:07,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:07,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:07,850 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,850 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:07,850 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:07,850 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:07,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:07,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:07,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:07,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:07,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:07,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:07,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66ca3bb3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:07,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:07,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:07,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:07,874 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,874 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:07,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:07,875 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:07,875 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:07,875 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:07,875 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:07,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:07,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:07,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:07,878 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:07,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:07,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:07,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:07,878 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:07,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:07,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:07,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:07,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:07,886 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703707870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703707872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703707889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703707889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:07,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:07,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:07,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:07,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:07,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:07,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:07,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:07,899 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:07,899 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:07,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703707893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,979 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417bcac7fe87d0345c6a00eb0047afec270 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bcac7fe87d0345c6a00eb0047afec270 2023-04-17 03:54:07,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703707991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703707992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703707992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:07,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703707994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:07,999 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:08,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:08,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:08,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703708001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:08,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:08,040 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:08,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:08,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:08,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,042 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,042 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:08,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:08,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:08,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:08,077 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:08,077 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,077 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:08,078 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:08,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:08,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:08,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:08,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:08,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71794f7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:08,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:08,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:08,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:08,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:08,103 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:08,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:08,108 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:08,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:08,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:08,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:08,108 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:08,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,111 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,113 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:08,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:08,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:08,117 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:08,117 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:08,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703708195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703708197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703708199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703708199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703708207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,218 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:08,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:08,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:08,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:08,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:08,256 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:08,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:08,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:08,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:08,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:08,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:08,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:08,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:08,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:08,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:08,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:08,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f36a2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:08,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:08,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:08,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:08,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:08,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:08,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:08,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:08,279 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:08,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:08,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:08,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:08,279 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:08,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,283 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:08,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:08,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:08,290 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:08,290 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:08,390 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:08,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:08,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:08,419 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:54:08,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:08,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:08,421 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:08,421 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,421 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:08,421 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:08,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:08,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:08,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:08,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:08,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:08,441 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,441 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:08,441 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:08,441 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:08,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:08,445 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:08,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c23cacb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:08,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:08,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:08,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:08,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,452 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:08,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:08,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:08,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:08,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:08,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:08,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:08,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,469 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,469 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:08,469 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,469 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:08,493 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/d79833c70a2f4875a59eac6f772c7341 2023-04-17 03:54:08,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703708499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703708499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703708501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703708506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:08,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703708511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,570 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:08,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e6043b43fa144e3a90f74f3f0376f332 2023-04-17 03:54:08,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:08,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:08,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:08,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:08,640 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:08,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:08,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:08,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:08,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:08,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:08,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:08,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:08,732 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:08,744 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,744 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:08,744 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:08,744 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,744 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:08,744 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:08,744 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,748 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:08,748 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:08,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/37ede0103c394a7da6d7083d64323199 2023-04-17 03:54:08,750 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:08,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b03755f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:08,752 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:08,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,754 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:08,754 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:08,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/d79833c70a2f4875a59eac6f772c7341 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341 2023-04-17 03:54:08,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:08,760 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,760 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:08,760 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,760 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:08,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341, entries=100, sequenceid=17, filesize=18.4 K 2023-04-17 03:54:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:08,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:08,771 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:08,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:08,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:08,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:08,771 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:08,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e6043b43fa144e3a90f74f3f0376f332 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332 2023-04-17 03:54:08,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:08,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,775 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:08,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:08,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:08,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:08,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:08,778 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:08,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,778 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:08,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:08,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:08,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:08,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332, entries=100, sequenceid=17, filesize=9.3 K 2023-04-17 03:54:08,820 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/37ede0103c394a7da6d7083d64323199 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199 2023-04-17 03:54:08,830 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199, entries=100, sequenceid=17, filesize=9.3 K 2023-04-17 03:54:08,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c6ba4e4e990f87093f177b6844dd61b9 in 997ms, sequenceid=17, compaction requested=false 2023-04-17 03:54:08,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:08,879 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:08,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:08,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:08,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:08,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:08,892 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:08,892 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,892 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:08,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:08,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,896 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:08,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:08,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:08,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:08,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:08,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:08,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:08,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:08,899 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:08,904 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:08,904 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:08,904 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:08,904 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:08,904 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:08,905 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:08,941 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417c4471df3a6a34358b6b3c00f8599dc34 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c4471df3a6a34358b6b3c00f8599dc34 2023-04-17 03:54:08,960 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/0f13d15ef7fc4919bbfe6fa186d72163 2023-04-17 03:54:09,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:09,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:09,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703709028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,387 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/d53f4ced92294d6c8b57c3277538822f 2023-04-17 03:54:09,421 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/35e05702e96140aeb9aa2c0e16d15bb6 2023-04-17 03:54:09,428 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/0f13d15ef7fc4919bbfe6fa186d72163 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163 2023-04-17 03:54:09,433 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163, entries=150, sequenceid=36, filesize=25.1 K 2023-04-17 03:54:09,434 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/d53f4ced92294d6c8b57c3277538822f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f 2023-04-17 03:54:09,449 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 03:54:09,450 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/35e05702e96140aeb9aa2c0e16d15bb6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6 2023-04-17 03:54:09,462 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6, entries=150, sequenceid=36, filesize=11.6 K 2023-04-17 03:54:09,463 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c6ba4e4e990f87093f177b6844dd61b9 in 559ms, sequenceid=36, compaction requested=false 2023-04-17 03:54:09,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:09,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:09,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:09,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:09,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:09,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:09,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5088e4e4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:09,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:09,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:09,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:09,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:09,474 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:09,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:09,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:09,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:09,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:09,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:09,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:09,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:09,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:09,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:09,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:09,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:09,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:09,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:09,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:09,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:09,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:09,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:09,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:09,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:09,493 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:09,493 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:09,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:09,593 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:09,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:09,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:09,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:09,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:09,597 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:09,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:09,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:09,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:09,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:09,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:09,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:09,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:09,600 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,600 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:09,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:09,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:09,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:09,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:09,601 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:09,601 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:09,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:09,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:09,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:09,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:09,603 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:09,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:09,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:09,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703709955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703709962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703709963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:09,965 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:09,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703709963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703710037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,055 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304174c137d1c15894967a3e17623f6eb78b9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174c137d1c15894967a3e17623f6eb78b9 2023-04-17 03:54:10,061 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/ffbab29f20b948feaa611c94c78253a7 2023-04-17 03:54:10,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703710259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703710267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703710267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703710267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,473 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/636aae7f6f9943089f63986c709c2d22 2023-04-17 03:54:10,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703710763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703710770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703710770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:10,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703710774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,907 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/67e1615ac27e446ab98c3570d0febe28 2023-04-17 03:54:10,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/ffbab29f20b948feaa611c94c78253a7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7 2023-04-17 03:54:10,917 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7, entries=150, sequenceid=50, filesize=25.1 K 2023-04-17 03:54:10,918 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/636aae7f6f9943089f63986c709c2d22 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22 2023-04-17 03:54:10,922 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 03:54:10,923 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/67e1615ac27e446ab98c3570d0febe28 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28 2023-04-17 03:54:10,927 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28, entries=150, sequenceid=50, filesize=11.6 K 2023-04-17 03:54:10,928 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for c6ba4e4e990f87093f177b6844dd61b9 in 1325ms, sequenceid=50, compaction requested=true 2023-04-17 03:54:10,928 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:10,928 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:10,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:10,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:10,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:10,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:10,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:10,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:10,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3d0271[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:10,933 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:10,933 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,934 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:10,935 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:10,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:10,937 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:10,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:10,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:10,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:10,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:10,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:10,941 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:10,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:10,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:10,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:10,941 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:10,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:10,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:10,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:10,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:10,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:10,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:10,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:10,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:10,949 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:10,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:10,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:10,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:10,949 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:10,949 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:10,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:10,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,049 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:11,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:11,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:11,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:11,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:11,053 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:11,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:11,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:11,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,055 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:11,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:11,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:11,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:11,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:11,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:11,057 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:54:11,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:11,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:11,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:11,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,072 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041761597040f3f2488c8f12db33ffca56f5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041761597040f3f2488c8f12db33ffca56f5 2023-04-17 03:54:11,079 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/7b5a0e9d52354eeab3c6a754d15b8f7f 2023-04-17 03:54:11,098 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e2914e8a572c4fcda635fcafa5c8dd7e 2023-04-17 03:54:11,520 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/120e9235a63d44f0b090790180ba08fd 2023-04-17 03:54:11,526 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/7b5a0e9d52354eeab3c6a754d15b8f7f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f 2023-04-17 03:54:11,531 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f, entries=150, sequenceid=69, filesize=25.1 K 2023-04-17 03:54:11,532 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e2914e8a572c4fcda635fcafa5c8dd7e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e 2023-04-17 03:54:11,537 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e, entries=150, sequenceid=69, filesize=11.6 K 2023-04-17 03:54:11,537 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/120e9235a63d44f0b090790180ba08fd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd 2023-04-17 03:54:11,547 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd, entries=150, sequenceid=69, filesize=11.6 K 2023-04-17 03:54:11,548 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for c6ba4e4e990f87093f177b6844dd61b9 in 491ms, sequenceid=69, compaction requested=true 2023-04-17 03:54:11,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:11,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:11,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:11,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:11,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:11,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:11,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0ae0cf[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:11,563 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:11,563 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:11,566 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,566 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:11,567 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:11,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:11,570 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:11,570 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:11,570 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:11,570 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:11,570 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:11,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:11,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:11,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:11,584 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,584 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:11,585 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,585 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,685 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:11,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:11,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:11,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:11,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:11,690 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:11,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:11,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:11,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,692 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:11,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:11,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:11,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:11,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:11,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:11,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:11,705 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,705 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:11,705 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:11,705 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:11,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:11,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:11,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:11,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,707 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,708 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,708 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:11,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dd3c9bc[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:11,710 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:11,710 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:11,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:11,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:11,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:11,714 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:11,714 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,714 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:11,714 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:11,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:11,720 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:11,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:11,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:11,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:11,720 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:11,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,727 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:11,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:11,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,728 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,728 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,728 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,728 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,728 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:11,728 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:11,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:11,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:11,828 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:11,828 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f188e4febffb45d9a77a4b324b22de9f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f188e4febffb45d9a77a4b324b22de9f 2023-04-17 03:54:11,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:11,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:11,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:11,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:11,848 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:11,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:11,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:11,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:11,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:11,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:11,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:11,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:11,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:11,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:11,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:11,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:11,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681703651837, max timestamp=1681703651855Num uniques 859; 2023-04-17 03:54:11,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681703651837, max timestamp=1681703651855Num uniques 859; 2023-04-17 03:54:11,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:11,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:11,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681703651837, max timestamp=1681703651855Num uniques 859; 2023-04-17 03:54:11,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:11,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:11,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6ba126[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:11,861 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:11,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:11,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:11,864 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,864 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,865 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:11,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:11,869 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:11,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:11,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:11,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:11,869 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:11,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,874 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:11,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,878 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:11,878 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:11,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703711889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703711889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703711890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703711891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,979 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:11,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:11,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:11,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:11,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:11,983 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:11,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:11,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:11,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:11,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:11,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:11,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:11,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:11,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:11,986 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,986 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:11,986 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:11,986 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:11,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:11,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:11,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,988 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,988 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,988 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:11,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d820490[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:11,990 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:11,990 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,991 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:11,991 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:11,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:11,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703711993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:11,994 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:11,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:11,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703711994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703711994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:11,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:11,997 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:11,997 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:11,997 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:11,997 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:11,997 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:11,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:11,999 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:11,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:11,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:11,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703711997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:11,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,003 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,003 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:12,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:12,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:12,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,004 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:12,004 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:12,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,004 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703712051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,055 DEBUG [Thread-844] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:12,104 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:12,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:12,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:12,115 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:12,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:12,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:12,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:12,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:12,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:12,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:12,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:12,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:12,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:12,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:12,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,121 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,122 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,122 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b48977[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:12,123 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:12,123 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,124 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:12,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:12,126 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:12,128 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:12,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:12,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:12,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:12,129 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:12,131 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,131 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,131 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:12,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,133 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:12,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:12,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,135 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:12,136 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:12,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:12,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:12,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:12,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:12,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703712195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703712198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703712199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703712200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,236 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:12,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:12,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:12,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:12,240 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:12,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:12,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:12,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,242 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,242 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,242 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:12,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:12,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:12,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:12,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:12,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:12,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:12,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:12,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74a92014[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:12,254 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:12,254 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,255 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=80, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/29986fd0cec7497aa67c71e25f2d35de 2023-04-17 03:54:12,257 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:12,257 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:12,257 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,257 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:12,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:12,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:12,264 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:12,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:12,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:12,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:12,264 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:12,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:12,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:12,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:12,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:12,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:12,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:12,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:12,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:12,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:12,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:12,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:12,272 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:12,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,272 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:12,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:12,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:12,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:12,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/39d3987c84b14909a4b9ee1a68e2941f 2023-04-17 03:54:12,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/abf8e36cc4474149b4b68f98207023fb 2023-04-17 03:54:12,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/29986fd0cec7497aa67c71e25f2d35de as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de 2023-04-17 03:54:12,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de, entries=350, sequenceid=80, filesize=52.3 K 2023-04-17 03:54:12,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/39d3987c84b14909a4b9ee1a68e2941f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f 2023-04-17 03:54:12,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f, entries=100, sequenceid=80, filesize=9.3 K 2023-04-17 03:54:12,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/abf8e36cc4474149b4b68f98207023fb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb 2023-04-17 03:54:12,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb, entries=100, sequenceid=80, filesize=9.3 K 2023-04-17 03:54:12,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for c6ba4e4e990f87093f177b6844dd61b9 in 525ms, sequenceid=80, compaction requested=true 2023-04-17 03:54:12,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:12,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:12,338 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:12,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:12,338 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:12,339 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:12,342 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54742 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:12,342 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:12,343 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,343 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=53.5 K 2023-04-17 03:54:12,343 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 149542 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:12,343 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:12,343 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,343 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=146.0 K 2023-04-17 03:54:12,344 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e6043b43fa144e3a90f74f3f0376f332, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681703647835 2023-04-17 03:54:12,344 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d79833c70a2f4875a59eac6f772c7341, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681703647835 2023-04-17 03:54:12,345 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d53f4ced92294d6c8b57c3277538822f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681703647865 2023-04-17 03:54:12,348 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0f13d15ef7fc4919bbfe6fa186d72163, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681703647865 2023-04-17 03:54:12,348 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 636aae7f6f9943089f63986c709c2d22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681703649008 2023-04-17 03:54:12,348 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ffbab29f20b948feaa611c94c78253a7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681703649008 2023-04-17 03:54:12,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 7b5a0e9d52354eeab3c6a754d15b8f7f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681703649639 2023-04-17 03:54:12,349 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e2914e8a572c4fcda635fcafa5c8dd7e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681703649639 2023-04-17 03:54:12,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 29986fd0cec7497aa67c71e25f2d35de, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703651780 2023-04-17 03:54:12,350 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 39d3987c84b14909a4b9ee1a68e2941f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703651810 2023-04-17 03:54:12,365 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#124 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:12,372 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:12,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:12,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:12,376 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#123 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:12,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:12,384 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:12,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:12,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:12,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,387 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:12,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:12,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:12,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:12,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:12,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:12,396 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:12,396 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:12,396 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:12,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:12,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:12,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:12,400 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-17 03:54:12,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:12,402 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:12,403 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:12,403 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:12,403 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:12,403 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:12,501 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703712506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703712506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703712507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703712509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703712611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703712611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703712611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703712612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,800 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/dc0ef2ca8eee44fba438ae924dc351c1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/dc0ef2ca8eee44fba438ae924dc351c1 2023-04-17 03:54:12,805 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into dc0ef2ca8eee44fba438ae924dc351c1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:12,805 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:12,805 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=11, startTime=1681703652338; duration=0sec 2023-04-17 03:54:12,805 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:12,805 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:12,807 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 54742 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:12,807 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:12,807 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:12,807 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=53.5 K 2023-04-17 03:54:12,808 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 37ede0103c394a7da6d7083d64323199, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681703647835 2023-04-17 03:54:12,808 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 35e05702e96140aeb9aa2c0e16d15bb6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681703647865 2023-04-17 03:54:12,808 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 67e1615ac27e446ab98c3570d0febe28, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681703649008 2023-04-17 03:54:12,809 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 120e9235a63d44f0b090790180ba08fd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681703649639 2023-04-17 03:54:12,809 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting abf8e36cc4474149b4b68f98207023fb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703651810 2023-04-17 03:54:12,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703712814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703712815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,818 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#126 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:12,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703712816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:12,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703712816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:12,832 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417027b59af4a644ac59bb90f9579f0b697 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417027b59af4a644ac59bb90f9579f0b697 2023-04-17 03:54:12,849 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/47890f81cc3047f4b954213fc254b11d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/47890f81cc3047f4b954213fc254b11d 2023-04-17 03:54:12,857 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bd1bccd26b1a49499ceae3fa33c420eb 2023-04-17 03:54:12,862 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into 47890f81cc3047f4b954213fc254b11d(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:12,862 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:12,862 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=11, startTime=1681703652338; duration=0sec 2023-04-17 03:54:12,862 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:13,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703713118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703713120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703713122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703713123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,228 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/fbc88d28b608475fb984bf5f42222365 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fbc88d28b608475fb984bf5f42222365 2023-04-17 03:54:13,233 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into fbc88d28b608475fb984bf5f42222365(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:13,233 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:13,233 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=11, startTime=1681703652339; duration=0sec 2023-04-17 03:54:13,233 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:13,271 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/050df742015f41c9bba46a1203537d7d 2023-04-17 03:54:13,286 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/b900fb357f0144dfb463b99c85378a8c 2023-04-17 03:54:13,292 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bd1bccd26b1a49499ceae3fa33c420eb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb 2023-04-17 03:54:13,297 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb, entries=150, sequenceid=115, filesize=25.1 K 2023-04-17 03:54:13,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/050df742015f41c9bba46a1203537d7d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d 2023-04-17 03:54:13,307 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 03:54:13,308 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/b900fb357f0144dfb463b99c85378a8c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c 2023-04-17 03:54:13,312 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c, entries=150, sequenceid=115, filesize=11.6 K 2023-04-17 03:54:13,313 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for c6ba4e4e990f87093f177b6844dd61b9 in 913ms, sequenceid=115, compaction requested=false 2023-04-17 03:54:13,314 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:13,315 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:13,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:13,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,317 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:13,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:13,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@239cdd36[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:13,319 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:13,319 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,320 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,320 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:13,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:13,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:13,321 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,321 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:13,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:13,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:13,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:13,326 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:13,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:13,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:13,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:13,326 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:13,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:13,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:13,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:13,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:13,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:13,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:13,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:13,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:13,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:13,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:13,367 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:13,367 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:13,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:13,467 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:13,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:13,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:13,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:13,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:13,471 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:13,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:13,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:13,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:13,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:13,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:13,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:13,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:13,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:13,477 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:13,477 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:13,493 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041750deb30863d046c8ad64c6421f0499df to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041750deb30863d046c8ad64c6421f0499df 2023-04-17 03:54:13,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:13,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:13,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703653641, max timestamp=1681703653647Num uniques 859; 2023-04-17 03:54:13,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:13,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703653641, max timestamp=1681703653647Num uniques 859; 2023-04-17 03:54:13,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:13,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703653641, max timestamp=1681703653647Num uniques 859; 2023-04-17 03:54:13,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:13,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703713689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703713690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703713690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703713691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703713792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703713794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703713794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703713796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:13,900 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=128, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bff38602f1124c9b95232e7340628bef 2023-04-17 03:54:13,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:13,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703713995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703713999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703714000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703714002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703714300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703714302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703714303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703714305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,313 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/3e65b574e9a045f7a480b3d0b317b0d3 2023-04-17 03:54:14,729 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=128 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/6f8663bea77f495fa692725789c740e4 2023-04-17 03:54:14,736 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bff38602f1124c9b95232e7340628bef as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef 2023-04-17 03:54:14,742 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef, entries=150, sequenceid=128, filesize=25.1 K 2023-04-17 03:54:14,744 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/3e65b574e9a045f7a480b3d0b317b0d3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3 2023-04-17 03:54:14,751 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3, entries=150, sequenceid=128, filesize=11.6 K 2023-04-17 03:54:14,753 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/6f8663bea77f495fa692725789c740e4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4 2023-04-17 03:54:14,758 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4, entries=150, sequenceid=128, filesize=11.6 K 2023-04-17 03:54:14,759 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c6ba4e4e990f87093f177b6844dd61b9 in 1282ms, sequenceid=128, compaction requested=true 2023-04-17 03:54:14,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:14,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:14,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:14,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:14,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:14,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:14,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:14,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee7fb51[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:14,764 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:14,764 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,765 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:14,765 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,765 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:14,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:14,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:14,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:14,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:14,767 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:14,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:14,770 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:14,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:14,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:14,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:14,770 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:14,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:14,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,772 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:14,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:14,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,777 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:14,777 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:14,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:14,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:14,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-17 03:54:14,808 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:14,808 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:54:14,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:14,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:54:14,809 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:14,809 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:54:14,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:14,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703714813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703714813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703714814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703714814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,841 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041796657aefc7c447a58a79faf25feaf273 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796657aefc7c447a58a79faf25feaf273 2023-04-17 03:54:14,877 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:14,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:14,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:14,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:14,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:14,881 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:14,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:14,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:14,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:14,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:14,882 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:14,883 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,883 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:14,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:14,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:14,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:14,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:14,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:14,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:14,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:14,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:14,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:14,886 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:14,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4b2acb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:14,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:14,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,888 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:14,889 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:14,889 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:14,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:14,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:14,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:14,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:14,896 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:14,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:14,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:14,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:14,896 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:14,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:14,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:14,899 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:14,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,901 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:14,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:14,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:14,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:14,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:14,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:14,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:14,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:14,905 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:14,905 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:14,905 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:14,905 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:14,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:14,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:14,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703714919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703714919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703714922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:14,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:14,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703714922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,005 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,009 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,021 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10df16f6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,024 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,024 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,025 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,027 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,029 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,029 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,030 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,031 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,032 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,036 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,036 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:15,036 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,037 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,037 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,037 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703715125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703715125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703715127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703715127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,137 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,141 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,142 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,144 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,144 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,144 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,144 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,148 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556f1cb6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,151 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,154 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,154 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,157 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,160 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,160 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,160 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:15,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,161 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,161 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,256 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/189bae0eff0d49c89c78e91df9c85af5 2023-04-17 03:54:15,261 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,267 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14935839[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,274 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,274 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,276 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,276 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,280 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,280 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,282 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,283 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,285 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,286 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,286 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,286 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,386 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607a2512[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,399 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,399 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,400 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,400 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,404 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,404 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,415 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,415 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703715427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703715428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703715429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703715430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,515 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,519 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,525 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,525 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,527 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,527 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87ae74d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,535 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,535 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,535 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,535 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,535 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,542 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,542 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,545 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,545 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,545 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,550 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,550 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,650 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,656 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,660 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,660 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,660 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,660 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,661 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ddc058[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,665 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,665 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,667 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,667 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,667 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,667 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e26a5d71466a4dde903aeb65046b0f61 2023-04-17 03:54:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,669 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,669 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,672 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,672 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,675 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,678 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,681 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,681 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,681 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,681 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,781 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,784 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,787 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,788 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,788 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,791 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,791 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b81064a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,794 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,795 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,797 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,797 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,800 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,800 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,800 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,801 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,804 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:15,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,804 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,804 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,804 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,905 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:15,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:15,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:15,908 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:15,908 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,908 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:15,908 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:15,909 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,909 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,910 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:15,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:15,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:15,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:15,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:15,912 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:15,912 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:15,913 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a62e49d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:15,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:15,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:15,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:15,918 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:15,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:15,921 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:15,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:15,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:15,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:15,921 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:15,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:15,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:15,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,926 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:15,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:15,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:15,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:15,930 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:15,930 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:15,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:15,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:15,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:15,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:15,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703715933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703715934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703715936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:15,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703715939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,030 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,033 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,035 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,040 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fd3813[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,042 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,042 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,045 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,047 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,048 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,050 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,051 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:16,054 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,054 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,080 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/389ac38f056149c18b4f6094e76d145f 2023-04-17 03:54:16,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/189bae0eff0d49c89c78e91df9c85af5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5 2023-04-17 03:54:16,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5, entries=200, sequenceid=166, filesize=32.1 K 2023-04-17 03:54:16,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e26a5d71466a4dde903aeb65046b0f61 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61 2023-04-17 03:54:16,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:16,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61, entries=150, sequenceid=166, filesize=11.8 K 2023-04-17 03:54:16,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703716094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/389ac38f056149c18b4f6094e76d145f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f 2023-04-17 03:54:16,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f, entries=150, sequenceid=166, filesize=11.8 K 2023-04-17 03:54:16,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for c6ba4e4e990f87093f177b6844dd61b9 in 1292ms, sequenceid=166, compaction requested=true 2023-04-17 03:54:16,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:16,100 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:16,101 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:16,101 DEBUG [Thread-844] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:16,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:16,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:16,103 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:16,103 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:16,103 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,103 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/dc0ef2ca8eee44fba438ae924dc351c1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=46.7 K 2023-04-17 03:54:16,103 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110296 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:16,103 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:16,103 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting dc0ef2ca8eee44fba438ae924dc351c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703649649 2023-04-17 03:54:16,103 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,104 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/47890f81cc3047f4b954213fc254b11d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=107.7 K 2023-04-17 03:54:16,104 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 050df742015f41c9bba46a1203537d7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681703651885 2023-04-17 03:54:16,104 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 47890f81cc3047f4b954213fc254b11d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703649649 2023-04-17 03:54:16,104 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3e65b574e9a045f7a480b3d0b317b0d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681703652504 2023-04-17 03:54:16,105 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bd1bccd26b1a49499ceae3fa33c420eb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681703651885 2023-04-17 03:54:16,105 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e26a5d71466a4dde903aeb65046b0f61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653689 2023-04-17 03:54:16,105 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bff38602f1124c9b95232e7340628bef, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681703652504 2023-04-17 03:54:16,106 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 189bae0eff0d49c89c78e91df9c85af5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653688 2023-04-17 03:54:16,113 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:16,142 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#136 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:16,149 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/70c70c16533a4a5682d5cef7aa226302 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/70c70c16533a4a5682d5cef7aa226302 2023-04-17 03:54:16,154 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into 70c70c16533a4a5682d5cef7aa226302(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:16,154 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,154 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=12, startTime=1681703656100; duration=0sec 2023-04-17 03:54:16,154 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,154 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:16,154 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:16,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,156 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:16,157 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:16,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,157 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,157 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fbc88d28b608475fb984bf5f42222365, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=46.7 K 2023-04-17 03:54:16,157 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fbc88d28b608475fb984bf5f42222365, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1681703649649 2023-04-17 03:54:16,158 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b900fb357f0144dfb463b99c85378a8c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1681703651885 2023-04-17 03:54:16,158 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 6f8663bea77f495fa692725789c740e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=128, earliestPutTs=1681703652504 2023-04-17 03:54:16,159 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 389ac38f056149c18b4f6094e76d145f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653689 2023-04-17 03:54:16,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,160 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,163 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,170 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#137 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:16,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:16,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,181 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,182 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:16,183 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,204 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417679c6341698a4d33bb0283a3869299a9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417679c6341698a4d33bb0283a3869299a9 2023-04-17 03:54:16,214 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bb95cf41b4924bc6a4fb5c62758e3fc2 2023-04-17 03:54:16,224 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/2c077c1ae6e74e89a7ee5119c9e999a7 2023-04-17 03:54:16,250 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/5b1024ec8fe2438585ab477927af0ccd 2023-04-17 03:54:16,255 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bb95cf41b4924bc6a4fb5c62758e3fc2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2 2023-04-17 03:54:16,261 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2, entries=150, sequenceid=176, filesize=25.3 K 2023-04-17 03:54:16,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/2c077c1ae6e74e89a7ee5119c9e999a7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7 2023-04-17 03:54:16,267 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7, entries=150, sequenceid=176, filesize=11.8 K 2023-04-17 03:54:16,268 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/5b1024ec8fe2438585ab477927af0ccd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd 2023-04-17 03:54:16,285 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd, entries=150, sequenceid=176, filesize=11.8 K 2023-04-17 03:54:16,286 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for c6ba4e4e990f87093f177b6844dd61b9 in 104ms, sequenceid=176, compaction requested=false 2023-04-17 03:54:16,286 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,286 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,286 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee9eff4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,290 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,290 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,291 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,302 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,304 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,304 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,304 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:16,304 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,304 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,305 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,305 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,405 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,412 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,413 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,413 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:16,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,420 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d791b35[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,441 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,441 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,441 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,441 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,441 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,442 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,442 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:16,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,542 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,546 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,548 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,548 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,548 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,548 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:16,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,550 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,550 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,550 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,550 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,552 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20b2440b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,556 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,557 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,557 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,561 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,561 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,565 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,568 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:16,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,568 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,568 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,568 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,568 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,594 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/a1ec00a9d5614c81979542ba9b52ff35 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a1ec00a9d5614c81979542ba9b52ff35 2023-04-17 03:54:16,594 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/aab4fe09bca14afbbe679932b3fd68e0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aab4fe09bca14afbbe679932b3fd68e0 2023-04-17 03:54:16,601 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into aab4fe09bca14afbbe679932b3fd68e0(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:16,601 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into a1ec00a9d5614c81979542ba9b52ff35(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:16,601 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,601 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,601 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=12, startTime=1681703656102; duration=0sec 2023-04-17 03:54:16,601 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=12, startTime=1681703656100; duration=0sec 2023-04-17 03:54:16,602 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:16,602 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:16,669 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,672 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,673 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,673 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,676 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,679 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,679 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,679 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,684 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,684 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,684 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,684 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c23ddae[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,687 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,687 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,689 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,718 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,718 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,726 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,726 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,727 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,727 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,727 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,828 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:16,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:16,848 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:16,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:16,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:16,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:16,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:16,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:16,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:16,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:16,860 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,860 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:16,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:16,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:16,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:16,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:16,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:16,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:16,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:16,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d741591[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:16,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:16,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:16,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,869 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:16,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:16,872 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:16,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:16,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:16,872 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:16,872 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:16,874 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:16,874 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,874 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:16,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:16,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:16,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:16,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:16,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:16,887 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:16,887 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:16,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:16,888 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,888 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:16,888 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:16,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:16,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:16,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:16,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:16,964 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:16,987 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:16,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:16,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:16,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:16,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703716995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703656992, max timestamp=1681703656997Num uniques 859; 2023-04-17 03:54:17,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:54:17,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703656992, max timestamp=1681703656997Num uniques 859; 2023-04-17 03:54:17,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:54:17,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703656992, max timestamp=1681703656998Num uniques 859; 2023-04-17 03:54:17,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:54:17,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:17,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:17,008 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:17,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:17,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:17,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,009 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,010 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:17,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:17,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:17,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:17,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:17,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:17,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:17,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:17,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:17,016 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,016 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:17,016 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:17,017 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,017 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:17,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28b10fb9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:17,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:17,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,021 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:17,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:17,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:17,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:17,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:17,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:17,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:17,025 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:17,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:17,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:17,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:17,025 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:17,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,027 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,028 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703717024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703717025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703717029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:17,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:17,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,033 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:17,033 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:17,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:17,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703717101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703717130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,133 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:17,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703717131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703717131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:17,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:17,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:17,136 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:17,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:17,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:17,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,139 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:17,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:17,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:17,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:17,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:17,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:17,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:17,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:17,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:17,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:17,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a9184b8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:17,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:17,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,146 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:17,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:17,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:17,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:17,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,147 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:17,147 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:17,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:17,150 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:17,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:17,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:17,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:17,151 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:17,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,157 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:17,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:17,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:17,158 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:17,158 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:17,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,258 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:17,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:17,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:17,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:17,261 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:17,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:17,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:17,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,263 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:17,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:17,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:17,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:17,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:17,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:17,285 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:17,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8cdf4e9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:17,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:17,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,290 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:17,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:17,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:17,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:17,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:17,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:17,297 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:17,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:17,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:17,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:17,297 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:17,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,306 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,310 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703717305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,310 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:17,311 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:17,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,311 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703717334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703717335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703717335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,404 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a605af031e714312b73d24a5e1fb6515 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a605af031e714312b73d24a5e1fb6515 2023-04-17 03:54:17,411 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:17,416 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=193, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/86ad7d1fab21497fb6a11bbc4763c76f 2023-04-17 03:54:17,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:17,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:17,420 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:17,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:17,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:17,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,424 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:17,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:17,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:17,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:17,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:17,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:17,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:17,438 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:17,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/178554f4ce014878ae9dbbacbe98cc6b 2023-04-17 03:54:17,443 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,443 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f8b0fc[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:17,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:17,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:17,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:17,453 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:17,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:17,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:17,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:17,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:17,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:17,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:17,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:17,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:17,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:17,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:17,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:17,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:17,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,467 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:17,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:17,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:17,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:17,467 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:17,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:17,467 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:17,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/ffae4322566e46e2bb3d88865b8bfc01 2023-04-17 03:54:17,478 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/86ad7d1fab21497fb6a11bbc4763c76f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f 2023-04-17 03:54:17,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f, entries=400, sequenceid=193, filesize=59.4 K 2023-04-17 03:54:17,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/178554f4ce014878ae9dbbacbe98cc6b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b 2023-04-17 03:54:17,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 03:54:17,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/ffae4322566e46e2bb3d88865b8bfc01 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01 2023-04-17 03:54:17,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01, entries=150, sequenceid=193, filesize=11.8 K 2023-04-17 03:54:17,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c6ba4e4e990f87093f177b6844dd61b9 in 534ms, sequenceid=193, compaction requested=true 2023-04-17 03:54:17,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:17,498 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:17,498 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:17,499 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:17,499 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:17,499 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:17,499 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:17,499 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,499 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:17,500 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/70c70c16533a4a5682d5cef7aa226302, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=35.6 K 2023-04-17 03:54:17,500 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,500 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a1ec00a9d5614c81979542ba9b52ff35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=110.3 K 2023-04-17 03:54:17,500 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 70c70c16533a4a5682d5cef7aa226302, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653689 2023-04-17 03:54:17,500 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a1ec00a9d5614c81979542ba9b52ff35, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653689 2023-04-17 03:54:17,501 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2c077c1ae6e74e89a7ee5119c9e999a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681703654811 2023-04-17 03:54:17,501 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bb95cf41b4924bc6a4fb5c62758e3fc2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681703654811 2023-04-17 03:54:17,501 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 178554f4ce014878ae9dbbacbe98cc6b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:17,502 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 86ad7d1fab21497fb6a11bbc4763c76f, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:17,510 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#144 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:17,521 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#145 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:17,542 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/b5576e3db0694316a66a712dde342798 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b5576e3db0694316a66a712dde342798 2023-04-17 03:54:17,547 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into b5576e3db0694316a66a712dde342798(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:17,547 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:17,547 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=13, startTime=1681703657498; duration=0sec 2023-04-17 03:54:17,547 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:17,547 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:17,548 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:17,548 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:17,548 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,549 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aab4fe09bca14afbbe679932b3fd68e0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=35.6 K 2023-04-17 03:54:17,549 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting aab4fe09bca14afbbe679932b3fd68e0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681703653689 2023-04-17 03:54:17,549 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5b1024ec8fe2438585ab477927af0ccd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681703654811 2023-04-17 03:54:17,550 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ffae4322566e46e2bb3d88865b8bfc01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:17,559 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#146 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:17,567 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/7437dedf4a0a4d47a3e92d0eff01c3e8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/7437dedf4a0a4d47a3e92d0eff01c3e8 2023-04-17 03:54:17,567 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:17,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:17,574 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 7437dedf4a0a4d47a3e92d0eff01c3e8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:17,574 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:17,574 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=13, startTime=1681703657499; duration=0sec 2023-04-17 03:54:17,574 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:17,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:17,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:17,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:17,578 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:17,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:17,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:17,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,579 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:17,579 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:17,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:17,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:17,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:17,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:17,581 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:17,581 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:17,582 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:17,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:17,614 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:17,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703717634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703717640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703717640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703717642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703717736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:17,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:17,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703717943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,007 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417eb5e09246a7345b2b6ee545f8ead78a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eb5e09246a7345b2b6ee545f8ead78a4 2023-04-17 03:54:18,016 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/03b543198c214dabad2a313ddb68729d 2023-04-17 03:54:18,040 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/566a98d20d0048c7b4824d5c468ec2d6 2023-04-17 03:54:18,057 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/e80188a83075436a83e1af4713d934dd 2023-04-17 03:54:18,063 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/03b543198c214dabad2a313ddb68729d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d 2023-04-17 03:54:18,068 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d, entries=150, sequenceid=230, filesize=25.3 K 2023-04-17 03:54:18,069 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/566a98d20d0048c7b4824d5c468ec2d6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6 2023-04-17 03:54:18,074 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6, entries=150, sequenceid=230, filesize=11.8 K 2023-04-17 03:54:18,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/e80188a83075436a83e1af4713d934dd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd 2023-04-17 03:54:18,096 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd, entries=150, sequenceid=230, filesize=11.8 K 2023-04-17 03:54:18,097 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for c6ba4e4e990f87093f177b6844dd61b9 in 515ms, sequenceid=230, compaction requested=false 2023-04-17 03:54:18,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:18,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:18,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:18,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:18,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4594a73d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:18,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:18,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,107 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,107 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:18,108 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:18,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:18,109 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:18,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:18,112 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:18,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:18,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:18,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:18,112 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:18,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,120 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:18,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:18,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,120 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:18,120 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:18,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:54:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:18,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:18,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:18,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304178d9db80e869c4061b9e0224dc8fee6c2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178d9db80e869c4061b9e0224dc8fee6c2 2023-04-17 03:54:18,180 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=242, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/f624f839365141e892df5bbbf2e72cb6 2023-04-17 03:54:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36218 deadline: 1681703718202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703718202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36282 deadline: 1681703718204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/91ba9494b7ed4917889c8413d0bfdc75 2023-04-17 03:54:18,221 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:18,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:18,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:18,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:18,225 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:18,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:18,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:18,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:18,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:18,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:18,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:18,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:18,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:18,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:18,241 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,241 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:18,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:18,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:18,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:18,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:18,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:18,244 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,244 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703718245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:18,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53365f73[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:18,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:18,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:18,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:18,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:18,249 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:18,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:18,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,252 WARN [DataStreamer for file /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/827c60ac1a2d44818b0842865e6c2bd8] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-17 03:54:18,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/827c60ac1a2d44818b0842865e6c2bd8 2023-04-17 03:54:18,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,258 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/f624f839365141e892df5bbbf2e72cb6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6 2023-04-17 03:54:18,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:18,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:18,259 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:18,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:18,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:18,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:18,259 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:18,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6, entries=150, sequenceid=242, filesize=25.3 K 2023-04-17 03:54:18,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/91ba9494b7ed4917889c8413d0bfdc75 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75 2023-04-17 03:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:18,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:18,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,272 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:18,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,272 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:18,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75, entries=150, sequenceid=242, filesize=11.8 K 2023-04-17 03:54:18,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/827c60ac1a2d44818b0842865e6c2bd8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8 2023-04-17 03:54:18,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8, entries=150, sequenceid=242, filesize=11.8 K 2023-04-17 03:54:18,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c6ba4e4e990f87093f177b6844dd61b9 in 134ms, sequenceid=242, compaction requested=true 2023-04-17 03:54:18,280 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:18,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:18,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:18,280 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:54:18,280 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-17 03:54:18,281 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-17 03:54:18,281 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-17 03:54:18,281 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. because compaction request was cancelled 2023-04-17 03:54:18,281 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:18,282 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:18,282 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:18,283 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,283 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b5576e3db0694316a66a712dde342798, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=35.7 K 2023-04-17 03:54:18,283 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b5576e3db0694316a66a712dde342798, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:18,284 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 566a98d20d0048c7b4824d5c468ec2d6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681703657022 2023-04-17 03:54:18,285 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 91ba9494b7ed4917889c8413d0bfdc75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681703657626 2023-04-17 03:54:18,297 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:18,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703658187, max timestamp=1681703658310Num uniques 859; 2023-04-17 03:54:18,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703658187, max timestamp=1681703658310Num uniques 859; 2023-04-17 03:54:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703658187, max timestamp=1681703658310Num uniques 859; 2023-04-17 03:54:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:18,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:18,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:18,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:18,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703658332, max timestamp=1681703658351Num uniques 859; 2023-04-17 03:54:18,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703658332, max timestamp=1681703658351Num uniques 859; 2023-04-17 03:54:18,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:54:18,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703658332, max timestamp=1681703658351Num uniques 859; 2023-04-17 03:54:18,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:54:18,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:54:18,358 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ade643ffe3134fbf903520c5d09d7a10 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ade643ffe3134fbf903520c5d09d7a10 2023-04-17 03:54:18,372 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:18,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:18,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:18,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:18,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:18,380 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:18,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:18,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:18,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,382 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,382 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,382 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,382 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703718385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703718386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703718387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:18,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:18,404 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:18,404 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,404 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:18,404 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=267, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/7d11a348f0944770a5c84be02cda5eba 2023-04-17 03:54:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:18,410 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,410 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:18,410 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:18,410 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,410 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:18,410 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:18,411 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,411 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:18,411 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:18,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:18,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343b8340[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:18,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:18,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:18,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:18,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,423 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:18,427 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:18,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:18,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:18,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:18,427 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:18,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,429 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:18,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:18,430 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:18,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:18,434 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:18,434 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:18,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:18,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/9b543224cc374a8ba434a0569260cb02 2023-04-17 03:54:18,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/91ab911cdc1b42569d5b4745b521aab1 2023-04-17 03:54:18,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/7d11a348f0944770a5c84be02cda5eba as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba 2023-04-17 03:54:18,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba, entries=250, sequenceid=267, filesize=39.2 K 2023-04-17 03:54:18,485 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/9b543224cc374a8ba434a0569260cb02 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02 2023-04-17 03:54:18,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02, entries=150, sequenceid=267, filesize=11.9 K 2023-04-17 03:54:18,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/91ab911cdc1b42569d5b4745b521aab1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1 2023-04-17 03:54:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703718493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36298 deadline: 1681703718493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703718493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1, entries=150, sequenceid=267, filesize=11.9 K 2023-04-17 03:54:18,509 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for c6ba4e4e990f87093f177b6844dd61b9 in 190ms, sequenceid=267, compaction requested=true 2023-04-17 03:54:18,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:18,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:54:18,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 03:54:18,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-17 03:54:18,534 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:18,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:18,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:18,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:18,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:18,540 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:18,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:18,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:18,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,542 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:18,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:18,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:18,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:18,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:18,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:18,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:18,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:18,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:18,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:18,548 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-17 03:54:18,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:18,549 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,549 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:18,549 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,549 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:18,549 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:18,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:18,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:18,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703718717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703718718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703718718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,726 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/b39439a3d4044a09ae34d3f5e118468b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b39439a3d4044a09ae34d3f5e118468b 2023-04-17 03:54:18,731 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into b39439a3d4044a09ae34d3f5e118468b(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:18,731 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:18,731 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=13, startTime=1681703658280; duration=0sec 2023-04-17 03:54:18,731 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-17 03:54:18,731 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 3 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:18,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:18,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction 2023-04-17 03:54:18,732 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,732 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=89.8 K 2023-04-17 03:54:18,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 03b543198c214dabad2a313ddb68729d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230 2023-04-17 03:54:18,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f624f839365141e892df5bbbf2e72cb6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=242 2023-04-17 03:54:18,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7d11a348f0944770a5c84be02cda5eba, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=267 2023-04-17 03:54:18,749 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#158 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:18,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703718748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,814 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/bf1c1ff5792a4dd89179734f93f1f503 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bf1c1ff5792a4dd89179734f93f1f503 2023-04-17 03:54:18,819 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into bf1c1ff5792a4dd89179734f93f1f503(size=25.7 K), total size for store is 115.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 03:54:18,820 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:18,820 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=13, startTime=1681703657498; duration=1sec 2023-04-17 03:54:18,820 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-17 03:54:18,820 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:18,821 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:18,821 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:18,821 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:18,821 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/7437dedf4a0a4d47a3e92d0eff01c3e8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=47.6 K 2023-04-17 03:54:18,822 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 7437dedf4a0a4d47a3e92d0eff01c3e8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:18,822 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e80188a83075436a83e1af4713d934dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681703657022 2023-04-17 03:54:18,822 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 827c60ac1a2d44818b0842865e6c2bd8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681703657626 2023-04-17 03:54:18,823 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 91ab911cdc1b42569d5b4745b521aab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681703658200 2023-04-17 03:54:18,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703718821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703718823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703718823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:18,831 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:18,969 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171fb4b708bc8c4f6e90333dc37e39c37d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171fb4b708bc8c4f6e90333dc37e39c37d 2023-04-17 03:54:19,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703719026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703719027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703719028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,240 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/277e4c3b724444d2bdbfc6b6e924fbd6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/277e4c3b724444d2bdbfc6b6e924fbd6 2023-04-17 03:54:19,244 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 277e4c3b724444d2bdbfc6b6e924fbd6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:19,245 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:19,245 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=12, startTime=1681703658509; duration=0sec 2023-04-17 03:54:19,245 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:54:19,245 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. because compaction request was cancelled 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-17 03:54:19,246 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. because compaction request was cancelled 2023-04-17 03:54:19,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703719330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703719330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703719332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,375 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/19a36375df4c4a178d56e7ae65702468 2023-04-17 03:54:19,569 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/164cfd2380b84c89abaf168274dec478 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/164cfd2380b84c89abaf168274dec478 2023-04-17 03:54:19,574 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into 164cfd2380b84c89abaf168274dec478(size=25.5 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:19,574 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:19,574 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=10, startTime=1681703658509; duration=0sec 2023-04-17 03:54:19,574 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:19,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36252 deadline: 1681703719757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,787 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e66176420b0246b581863d163cfa4ab5 2023-04-17 03:54:19,798 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/10f411d76937440ca3de668a4b160a71 2023-04-17 03:54:19,803 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/19a36375df4c4a178d56e7ae65702468 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468 2023-04-17 03:54:19,807 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468, entries=150, sequenceid=302, filesize=25.4 K 2023-04-17 03:54:19,807 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e66176420b0246b581863d163cfa4ab5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5 2023-04-17 03:54:19,811 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5, entries=150, sequenceid=302, filesize=11.9 K 2023-04-17 03:54:19,812 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/10f411d76937440ca3de668a4b160a71 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71 2023-04-17 03:54:19,822 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71, entries=150, sequenceid=302, filesize=11.9 K 2023-04-17 03:54:19,823 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for c6ba4e4e990f87093f177b6844dd61b9 in 1275ms, sequenceid=302, compaction requested=true 2023-04-17 03:54:19,823 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:19,823 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:19,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:19,825 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:19,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:19,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1464aa6b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:19,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:19,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,828 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:19,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:19,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:19,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:19,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:19,832 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:19,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:19,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:19,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:19,832 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:19,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:19,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:19,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:19,836 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:19,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:19,837 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:19,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:19,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:19,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:19,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:19,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,840 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,840 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703659851, max timestamp=1681703659862Num uniques 859; 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703659851, max timestamp=1681703659862Num uniques 859; 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703659851, max timestamp=1681703659862Num uniques 859; 2023-04-17 03:54:19,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:54:19,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703719898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703719899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:19,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703719900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,941 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:19,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:19,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:19,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:19,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:19,944 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:19,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:19,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:19,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:19,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:19,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74294a95[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:19,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:19,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:19,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:19,956 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:19,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:19,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:19,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:19,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:19,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:19,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:19,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:19,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:19,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:19,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:19,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:19,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:19,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:19,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:19,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,964 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:19,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:19,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:19,964 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:19,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703720002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703720002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703720004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,065 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,068 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,073 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,073 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,073 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,074 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,074 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,074 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,074 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15055ea4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,081 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,083 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,083 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,088 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,088 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,088 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,093 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,093 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,193 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,196 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,199 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,200 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,200 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,200 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,200 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,200 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,202 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dccfec7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703720205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,208 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703720205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703720207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,212 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,213 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,214 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,214 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,214 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,218 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,219 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,250 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041743fa1276cc964fc7be6e47566ad5865a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041743fa1276cc964fc7be6e47566ad5865a 2023-04-17 03:54:20,319 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,322 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,323 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,323 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,324 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,326 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,326 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,326 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,326 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,328 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cca24e7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,330 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,334 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,335 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,335 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,335 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,335 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,343 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,347 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,347 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,447 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,450 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,451 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,455 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7209a209[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,463 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,463 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:20,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,472 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,472 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703720510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703720510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:20,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703720511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,572 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,576 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,580 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,580 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,581 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,582 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc62292[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,585 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,587 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,587 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,587 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,587 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,589 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,589 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,592 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:20,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,595 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,595 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,656 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/16e6b6b8c567490fb82e672f8bbc22da 2023-04-17 03:54:20,695 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,698 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,699 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,699 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,700 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,702 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,702 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,702 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,702 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5766d72b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,707 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,708 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,708 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,711 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,711 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,713 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,713 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,713 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,723 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:20,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,724 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,724 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,724 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,724 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,724 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,724 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,824 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,827 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,828 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,832 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a163c3c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,835 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,842 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,843 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,844 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,844 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,844 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,850 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,850 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,951 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:20,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:20,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:20,954 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:20,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:20,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:20,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,957 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:20,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:20,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:20,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:20,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:20,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:20,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7530c69d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:20,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:20,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:20,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:20,970 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:20,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:20,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:20,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:20,971 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:20,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:20,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,981 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:20,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:20,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:20,982 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:20,982 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,982 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:20,982 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:20,982 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:20,982 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:20,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:21,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:21,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703721015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703721017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:21,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703721018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/264d41d7185a4926a376976e6d511545 2023-04-17 03:54:21,082 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:21,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:21,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:21,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:21,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:21,085 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:21,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:21,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:21,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,087 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:21,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:21,089 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,089 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:21,089 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:21,089 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:21,091 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:21,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@99ea81b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:21,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:21,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:21,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:21,100 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:21,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:21,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:21,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:21,100 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:21,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:21,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:21,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,105 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:21,105 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:21,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,205 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:21,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:21,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:21,209 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:21,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:21,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:21,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:21,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a0f98c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:21,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:21,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:21,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:21,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:21,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:21,223 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:21,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:21,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:21,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:21,223 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:21,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,231 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:21,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:21,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:21,232 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:21,232 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:21,332 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:21,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:21,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:21,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:21,336 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:21,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:21,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:21,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:21,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:21,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:21,339 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,339 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:21,339 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:21,339 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:21,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:21,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:21,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13accf78[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:21,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:21,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:21,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,349 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:21,351 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:21,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:21,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:21,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:21,351 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:21,353 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,353 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,353 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,353 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:21,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:21,359 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:21,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,359 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:21,360 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:21,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,460 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:21,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:21,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:21,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:21,463 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:21,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:21,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:21,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:21,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:21,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:21,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:21,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:21,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:21,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:21,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:21,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:21,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605be7be[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:21,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:21,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:21,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:21,475 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:21,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:21,477 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:21,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:21,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:21,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:21,477 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:21,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/fb86186efab0426daae72c10651db5a7 2023-04-17 03:54:21,479 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,479 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,479 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/16e6b6b8c567490fb82e672f8bbc22da as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da 2023-04-17 03:54:21,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:21,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:21,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:21,485 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:21,485 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:21,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da, entries=200, sequenceid=317, filesize=32.3 K 2023-04-17 03:54:21,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/264d41d7185a4926a376976e6d511545 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545 2023-04-17 03:54:21,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545, entries=150, sequenceid=317, filesize=11.9 K 2023-04-17 03:54:21,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/fb86186efab0426daae72c10651db5a7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7 2023-04-17 03:54:21,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7, entries=150, sequenceid=317, filesize=11.9 K 2023-04-17 03:54:21,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for c6ba4e4e990f87093f177b6844dd61b9 in 1666ms, sequenceid=317, compaction requested=true 2023-04-17 03:54:21,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:21,503 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:21,503 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:21,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:21,504 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49106 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:21,505 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:21,505 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,505 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b39439a3d4044a09ae34d3f5e118468b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=48.0 K 2023-04-17 03:54:21,505 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111587 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:21,505 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:21,505 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,505 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bf1c1ff5792a4dd89179734f93f1f503, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/164cfd2380b84c89abaf168274dec478, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=109.0 K 2023-04-17 03:54:21,506 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b39439a3d4044a09ae34d3f5e118468b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1681703657626 2023-04-17 03:54:21,507 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9b543224cc374a8ba434a0569260cb02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681703658200 2023-04-17 03:54:21,507 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bf1c1ff5792a4dd89179734f93f1f503, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=193, earliestPutTs=1681703656951 2023-04-17 03:54:21,507 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e66176420b0246b581863d163cfa4ab5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681703658375 2023-04-17 03:54:21,508 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 164cfd2380b84c89abaf168274dec478, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681703658200 2023-04-17 03:54:21,508 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 264d41d7185a4926a376976e6d511545, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:21,509 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 19a36375df4c4a178d56e7ae65702468, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681703658375 2023-04-17 03:54:21,509 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 16e6b6b8c567490fb82e672f8bbc22da, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:21,522 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#165 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:21,545 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#166 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:21,585 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:21,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:21,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:21,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:21,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:21,589 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:21,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:21,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:21,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:21,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:21,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:21,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:21,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:21,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:21,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:21,593 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:21,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:21,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:21,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:21,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703721791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:21,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703721895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:21,944 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e9b7fbb0c172411ea0c90017f2a1f77d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b7fbb0c172411ea0c90017f2a1f77d 2023-04-17 03:54:21,949 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into e9b7fbb0c172411ea0c90017f2a1f77d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:21,949 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:21,949 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=12, startTime=1681703661503; duration=0sec 2023-04-17 03:54:21,949 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:21,949 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:21,950 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:21,950 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:21,950 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:21,950 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/277e4c3b724444d2bdbfc6b6e924fbd6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=36.2 K 2023-04-17 03:54:21,951 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 277e4c3b724444d2bdbfc6b6e924fbd6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1681703658200 2023-04-17 03:54:21,951 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 10f411d76937440ca3de668a4b160a71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681703658375 2023-04-17 03:54:21,952 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fb86186efab0426daae72c10651db5a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:21,958 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#168 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:21,963 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/f2fa264710a948ce983f90d5012396a5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f2fa264710a948ce983f90d5012396a5 2023-04-17 03:54:21,968 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into f2fa264710a948ce983f90d5012396a5(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:21,968 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:21,968 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=12, startTime=1681703661503; duration=0sec 2023-04-17 03:54:21,968 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:22,005 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041700b540637baf4545a33baafcb62abbe4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041700b540637baf4545a33baafcb62abbe4 2023-04-17 03:54:22,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703722018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703722022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703722031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703722099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,367 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/034e0bace0054231b8779e0e10ca15f0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/034e0bace0054231b8779e0e10ca15f0 2023-04-17 03:54:22,372 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 034e0bace0054231b8779e0e10ca15f0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:22,373 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:22,373 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=13, startTime=1681703661503; duration=0sec 2023-04-17 03:54:22,373 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:22,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703722401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,411 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=351, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/d044383f16a14d9b9901347d69150a6d 2023-04-17 03:54:22,422 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/a96b3e0047ed4df19674ae470b0638da 2023-04-17 03:54:22,836 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/0862a28b2b5940ba90e4e489892bf39e 2023-04-17 03:54:22,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/d044383f16a14d9b9901347d69150a6d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d 2023-04-17 03:54:22,845 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d, entries=150, sequenceid=351, filesize=25.4 K 2023-04-17 03:54:22,846 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/a96b3e0047ed4df19674ae470b0638da as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da 2023-04-17 03:54:22,850 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da, entries=150, sequenceid=351, filesize=11.9 K 2023-04-17 03:54:22,851 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/0862a28b2b5940ba90e4e489892bf39e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e 2023-04-17 03:54:22,855 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e, entries=150, sequenceid=351, filesize=11.9 K 2023-04-17 03:54:22,856 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for c6ba4e4e990f87093f177b6844dd61b9 in 1263ms, sequenceid=351, compaction requested=false 2023-04-17 03:54:22,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:22,856 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:22,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:22,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:22,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7955de[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:22,861 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:22,861 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:22,862 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:22,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:22,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:22,864 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:22,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:22,866 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:22,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:22,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:22,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:22,867 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:22,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:22,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:22,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:22,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:22,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,870 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:22,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:22,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:22,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:22,873 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:22,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:22,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:22,874 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:22,874 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:22,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:22,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:22,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:22,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:22,910 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:22,938 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417002750685b4a409eb2947516260d13cc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417002750685b4a409eb2947516260d13cc 2023-04-17 03:54:22,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:22,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703722958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,974 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:22,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:22,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:22,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:22,978 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:22,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:22,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:22,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:22,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:22,979 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:22,980 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:22,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:22,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:22,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:22,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:22,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:22,982 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:22,982 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:22,982 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:22,982 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:22,982 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:22,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,984 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:22,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:22,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73faa344[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:22,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:22,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,996 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:22,996 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,996 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:22,997 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:22,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:22,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,000 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,000 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,002 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,002 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,002 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,005 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,006 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,006 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:23,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703723061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,106 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,109 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,109 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,109 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,109 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,114 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,114 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,114 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,114 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,114 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,115 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,116 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658654c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,119 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,122 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,122 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,124 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,124 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,124 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,128 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,128 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,228 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,234 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,234 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,238 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,239 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,239 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,239 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,239 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,240 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3901042e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,242 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,244 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,245 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,247 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,247 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,252 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,253 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:23,253 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703723263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,357 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/8e5f19c2ca7941e0986f70feea8cf8d5 2023-04-17 03:54:23,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,361 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,364 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,365 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,365 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,365 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,365 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,365 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,370 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f006175[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,372 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,379 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,379 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,382 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,386 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,386 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,386 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,486 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,489 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,492 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,497 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c60f52[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,497 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,499 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,501 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,508 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,508 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,567 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:23,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703723566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,608 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,611 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,611 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,611 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,611 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,613 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,613 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,614 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,614 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,617 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cf17123[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,626 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,626 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,626 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,627 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,629 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,629 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,630 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,630 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,730 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,734 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,734 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,734 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,734 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,734 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,734 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,737 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,737 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,737 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:23,737 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:23,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:23,739 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,739 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:23,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6024e7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:23,741 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:23,741 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,742 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:23,742 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,742 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:23,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:23,743 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:23,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:23,745 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:23,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:23,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:23,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:23,746 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:23,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:23,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,747 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:23,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:23,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,748 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:23,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:23,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:23,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:23,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:23,758 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:23,758 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:23,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:23,772 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/7c70daa50d1f4ea4a51a02c8365e6ccc 2023-04-17 03:54:23,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/aac2140e7c144b0c8fd174bd87ab147f 2023-04-17 03:54:23,787 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/8e5f19c2ca7941e0986f70feea8cf8d5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5 2023-04-17 03:54:23,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5, entries=200, sequenceid=366, filesize=32.3 K 2023-04-17 03:54:23,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/7c70daa50d1f4ea4a51a02c8365e6ccc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc 2023-04-17 03:54:23,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc, entries=150, sequenceid=366, filesize=11.9 K 2023-04-17 03:54:23,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/aac2140e7c144b0c8fd174bd87ab147f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f 2023-04-17 03:54:23,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f, entries=150, sequenceid=366, filesize=11.9 K 2023-04-17 03:54:23,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c6ba4e4e990f87093f177b6844dd61b9 in 905ms, sequenceid=366, compaction requested=true 2023-04-17 03:54:23,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:23,814 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:23,814 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:23,815 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:23,815 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:23,816 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,816 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:23,816 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b7fbb0c172411ea0c90017f2a1f77d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=36.3 K 2023-04-17 03:54:23,816 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:23,816 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,816 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f2fa264710a948ce983f90d5012396a5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=83.8 K 2023-04-17 03:54:23,816 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e9b7fbb0c172411ea0c90017f2a1f77d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:23,817 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f2fa264710a948ce983f90d5012396a5, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:23,817 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting a96b3e0047ed4df19674ae470b0638da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681703659894 2023-04-17 03:54:23,817 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d044383f16a14d9b9901347d69150a6d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681703659894 2023-04-17 03:54:23,817 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7c70daa50d1f4ea4a51a02c8365e6ccc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:23,817 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8e5f19c2ca7941e0986f70feea8cf8d5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:23,825 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:23,839 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#175 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:23,851 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/842e1b38111144c2b9e74f5b4ad34970 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/842e1b38111144c2b9e74f5b4ad34970 2023-04-17 03:54:23,858 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into 842e1b38111144c2b9e74f5b4ad34970(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:23,858 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:23,858 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=13, startTime=1681703663814; duration=0sec 2023-04-17 03:54:23,858 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:23,858 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:23,858 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:23,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:23,859 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:23,859 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:23,859 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,860 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/034e0bace0054231b8779e0e10ca15f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=36.3 K 2023-04-17 03:54:23,860 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 034e0bace0054231b8779e0e10ca15f0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681703658715 2023-04-17 03:54:23,861 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0862a28b2b5940ba90e4e489892bf39e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1681703659894 2023-04-17 03:54:23,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:23,861 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting aac2140e7c144b0c8fd174bd87ab147f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:23,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:23,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:23,868 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:23,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:23,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:23,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,870 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:23,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:23,870 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:23,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:23,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:23,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:23,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:23,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:23,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:23,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:23,876 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:23,876 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:23,884 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:23,884 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:23,897 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/1d43304307fd4d51aa926e5701b20cdb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/1d43304307fd4d51aa926e5701b20cdb 2023-04-17 03:54:23,909 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 1d43304307fd4d51aa926e5701b20cdb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:23,909 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:23,909 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=13, startTime=1681703663814; duration=0sec 2023-04-17 03:54:23,909 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:23,928 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417698d00ac1d234f4ea853859a53964981 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417698d00ac1d234f4ea853859a53964981 2023-04-17 03:54:23,946 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=387, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/655f3e58c18a45a88858be0ad0b1d827 2023-04-17 03:54:23,971 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e9b42a5184ea4cf488ec92d14a7efbe3 2023-04-17 03:54:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:24,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:24,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703724068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,277 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/e94a3c1472844fabb6e45a35d96ce8ab as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/e94a3c1472844fabb6e45a35d96ce8ab 2023-04-17 03:54:24,285 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into e94a3c1472844fabb6e45a35d96ce8ab(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:24,286 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:24,286 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=13, startTime=1681703663814; duration=0sec 2023-04-17 03:54:24,286 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:24,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,387 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=387 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/5641175241f74a45a04ae867006c7362 2023-04-17 03:54:24,393 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/655f3e58c18a45a88858be0ad0b1d827 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827 2023-04-17 03:54:24,398 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827, entries=150, sequenceid=387, filesize=25.4 K 2023-04-17 03:54:24,399 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/e9b42a5184ea4cf488ec92d14a7efbe3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3 2023-04-17 03:54:24,403 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3, entries=150, sequenceid=387, filesize=11.9 K 2023-04-17 03:54:24,404 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/5641175241f74a45a04ae867006c7362 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362 2023-04-17 03:54:24,413 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362, entries=150, sequenceid=387, filesize=11.9 K 2023-04-17 03:54:24,414 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for c6ba4e4e990f87093f177b6844dd61b9 in 530ms, sequenceid=387, compaction requested=false 2023-04-17 03:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:24,414 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:24,414 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:24,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:24,417 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,417 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5107f1f7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:24,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:24,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:24,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:24,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:24,427 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:24,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:24,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:24,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:24,427 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:24,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:24,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:24,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:24,429 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,433 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:24,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:24,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:24,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:24,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:24,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,435 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:24,435 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:24,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:24,535 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:24,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:24,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:24,539 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:24,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:24,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:24,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:24,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:24,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:24,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:24,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:24,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:24,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:24,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:24,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:24,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:24,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:24,543 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:24,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:24,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:24,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,964 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177c6922746edf4b4ab1b94e3ba4a03cb0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177c6922746edf4b4ab1b94e3ba4a03cb0 2023-04-17 03:54:24,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703724977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703724977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:24,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703724981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703725077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703725280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703725281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703725283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,370 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=402, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/1cdc73c4ccc94fb8ba9c893808b7b839 2023-04-17 03:54:25,781 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/4efe196639eb49abacf12403d52b077c 2023-04-17 03:54:25,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703725783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703725786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:25,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703725790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36228 deadline: 1681703726112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,115 DEBUG [Thread-844] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:26,203 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/8a7acb9685af48d894cdf66468dc8f72 2023-04-17 03:54:26,211 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/1cdc73c4ccc94fb8ba9c893808b7b839 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839 2023-04-17 03:54:26,215 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839, entries=150, sequenceid=402, filesize=25.4 K 2023-04-17 03:54:26,216 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/4efe196639eb49abacf12403d52b077c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c 2023-04-17 03:54:26,220 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c, entries=150, sequenceid=402, filesize=11.9 K 2023-04-17 03:54:26,221 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/8a7acb9685af48d894cdf66468dc8f72 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72 2023-04-17 03:54:26,225 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72, entries=150, sequenceid=402, filesize=11.9 K 2023-04-17 03:54:26,225 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c6ba4e4e990f87093f177b6844dd61b9 in 1682ms, sequenceid=402, compaction requested=true 2023-04-17 03:54:26,226 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:26,226 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c2c81a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,231 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,236 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,236 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,238 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,243 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,244 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,244 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,344 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:26,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:26,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:26,347 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:26,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:26,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:26,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:26,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:26,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:26,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:26,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:26,352 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:26,352 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,368 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041785c46a1b111246a5bda5d21f6236ba0d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041785c46a1b111246a5bda5d21f6236ba0d 2023-04-17 03:54:26,376 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=421, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/4f582d17805040638ab9c1772c7d6c1c 2023-04-17 03:54:26,388 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/fe8f6a823d9344f580b9d7bb2cab53c9 2023-04-17 03:54:26,397 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/e934103aece74c11a2ca26c5ab1281cb 2023-04-17 03:54:26,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/4f582d17805040638ab9c1772c7d6c1c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c 2023-04-17 03:54:26,405 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c, entries=150, sequenceid=421, filesize=25.4 K 2023-04-17 03:54:26,405 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/fe8f6a823d9344f580b9d7bb2cab53c9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9 2023-04-17 03:54:26,410 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9, entries=150, sequenceid=421, filesize=11.9 K 2023-04-17 03:54:26,410 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/e934103aece74c11a2ca26c5ab1281cb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb 2023-04-17 03:54:26,420 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb, entries=150, sequenceid=421, filesize=11.9 K 2023-04-17 03:54:26,421 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for c6ba4e4e990f87093f177b6844dd61b9 in 69ms, sequenceid=421, compaction requested=true 2023-04-17 03:54:26,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:26,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,423 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3dcc48[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,440 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,448 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:26,448 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,449 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,449 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,449 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,449 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,553 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:26,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:26,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:26,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:26,558 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:26,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:26,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:26,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,560 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:26,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:26,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:26,562 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,562 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:26,562 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:26,562 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,566 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,566 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,566 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c69254[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,571 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,574 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,574 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,576 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,576 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,576 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,582 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,583 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,683 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:26,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:26,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:26,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:26,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:26,688 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:26,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:26,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:26,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,691 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,691 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:26,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:26,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:26,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:26,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:26,701 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:26,701 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,703 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7da04192[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,706 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,712 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,712 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,712 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,712 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,712 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,714 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,714 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,714 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,714 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,719 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,719 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:26,819 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:26,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:26,825 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:26,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:26,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:26,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,827 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:26,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:26,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:26,829 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:26,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:26,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:26,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:26,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:26,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:26,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:26,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:26,830 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439cd424[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,837 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,842 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,843 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,848 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,852 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,853 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:26,855 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,855 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,861 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417034df6c14b0c49e9850059c4a6f06939 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417034df6c14b0c49e9850059c4a6f06939 2023-04-17 03:54:26,875 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/ec46a0ea29554ac8b46060eca7aad54e 2023-04-17 03:54:26,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681703666833, max timestamp=1681703666879Num uniques 859; 2023-04-17 03:54:26,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 03:54:26,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681703666833, max timestamp=1681703666879Num uniques 859; 2023-04-17 03:54:26,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 03:54:26,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1681703666833, max timestamp=1681703666879Num uniques 859; 2023-04-17 03:54:26,881 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-17 03:54:26,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703726888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703726890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703726891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,955 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:26,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:26,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:26,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:26,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:26,959 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:26,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:26,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:26,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:26,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:26,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:26,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:26,967 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:26,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:26,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:26,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:26,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:26,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:26,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:26,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2021f93[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:26,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:26,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:26,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:26,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:26,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:26,979 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:26,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:26,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:26,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:26,980 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:26,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:26,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:26,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:26,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:26,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:26,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:26,989 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:26,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:26,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:26,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:26,990 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:26,990 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:26,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703726992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703726993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:26,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:26,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703726996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,090 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:27,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:27,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:27,093 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:27,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:27,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:27,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36252 deadline: 1681703727091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,096 DEBUG [Thread-846] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:27,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:27,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:27,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:27,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:27,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:27,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:27,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:27,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:27,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:27,099 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:27,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:27,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25f179b5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:27,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:27,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,103 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:27,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:27,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:27,104 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:27,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:27,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:27,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:27,107 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:27,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:27,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:27,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:27,107 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:27,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,111 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:27,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:27,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:27,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:27,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:27,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:27,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,116 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:27,116 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:27,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703727195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703727196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703727198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,216 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:27,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:27,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:27,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:27,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:27,219 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:27,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:27,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:27,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:27,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:27,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:27,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:27,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:27,223 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,223 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:27,223 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:27,223 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:27,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:27,224 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,225 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:27,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:27,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a221048[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:27,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:27,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:27,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:27,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:27,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:27,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:27,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:27,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:27,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:27,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:27,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:27,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:27,235 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:27,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:27,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:27,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:27,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:27,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:27,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,239 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:27,239 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:27,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:27,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/afa2001aad91409e8989060cca154a7e 2023-04-17 03:54:27,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/6a6b46ee0eae4705b457d20cccff00d5 2023-04-17 03:54:27,325 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/ec46a0ea29554ac8b46060eca7aad54e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e 2023-04-17 03:54:27,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e, entries=350, sequenceid=433, filesize=52.9 K 2023-04-17 03:54:27,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/afa2001aad91409e8989060cca154a7e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e 2023-04-17 03:54:27,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e, entries=350, sequenceid=433, filesize=21.4 K 2023-04-17 03:54:27,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/6a6b46ee0eae4705b457d20cccff00d5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5 2023-04-17 03:54:27,339 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:27,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5, entries=150, sequenceid=433, filesize=11.9 K 2023-04-17 03:54:27,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for c6ba4e4e990f87093f177b6844dd61b9 in 513ms, sequenceid=433, compaction requested=true 2023-04-17 03:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:27,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:27,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:54:27,342 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:27,342 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:27,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:27,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:27,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:27,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:27,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:27,345 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 159148 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:27,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,345 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:27,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,345 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,345 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/e94a3c1472844fabb6e45a35d96ce8ab, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=155.4 K 2023-04-17 03:54:27,345 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 71444 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:27,345 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:27,345 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,345 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/842e1b38111144c2b9e74f5b4ad34970, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=69.8 K 2023-04-17 03:54:27,346 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e94a3c1472844fabb6e45a35d96ce8ab, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:27,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:27,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:27,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:27,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:27,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 842e1b38111144c2b9e74f5b4ad34970, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:27,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e9b42a5184ea4cf488ec92d14a7efbe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681703662938 2023-04-17 03:54:27,349 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 655f3e58c18a45a88858be0ad0b1d827, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681703662938 2023-04-17 03:54:27,350 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 4efe196639eb49abacf12403d52b077c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681703664032 2023-04-17 03:54:27,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:27,356 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting fe8f6a823d9344f580b9d7bb2cab53c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681703664663 2023-04-17 03:54:27,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:27,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:27,356 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting afa2001aad91409e8989060cca154a7e, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:27,357 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1cdc73c4ccc94fb8ba9c893808b7b839, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681703664032 2023-04-17 03:54:27,357 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4f582d17805040638ab9c1772c7d6c1c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681703664663 2023-04-17 03:54:27,358 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ec46a0ea29554ac8b46060eca7aad54e, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:27,361 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:27,361 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,361 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. started... 2023-04-17 03:54:27,361 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:27,362 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:27,371 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#189 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:27,394 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#190 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:27,400 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/3bb9fadc70aa4784aeb3dc1919b8d982 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3bb9fadc70aa4784aeb3dc1919b8d982 2023-04-17 03:54:27,408 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into 3bb9fadc70aa4784aeb3dc1919b8d982(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:27,408 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:27,408 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=11, startTime=1681703667342; duration=0sec 2023-04-17 03:54:27,408 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:27,408 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:27,410 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:27,410 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:27,411 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:27,411 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/1d43304307fd4d51aa926e5701b20cdb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=60.2 K 2023-04-17 03:54:27,411 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1d43304307fd4d51aa926e5701b20cdb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1681703661772 2023-04-17 03:54:27,411 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5641175241f74a45a04ae867006c7362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=387, earliestPutTs=1681703662938 2023-04-17 03:54:27,412 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8a7acb9685af48d894cdf66468dc8f72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1681703664032 2023-04-17 03:54:27,412 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e934103aece74c11a2ca26c5ab1281cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1681703664663 2023-04-17 03:54:27,412 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6a6b46ee0eae4705b457d20cccff00d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:27,424 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#192 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:27,438 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/35634699ac7642cab6e122195dc9a348 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35634699ac7642cab6e122195dc9a348 2023-04-17 03:54:27,443 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 35634699ac7642cab6e122195dc9a348(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:27,443 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:27,443 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=11, startTime=1681703667342; duration=0sec 2023-04-17 03:54:27,443 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:27,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:27,501 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. as already flushing 2023-04-17 03:54:27,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703727508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703727509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703727509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703727612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703727612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703727613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,796 DEBUG [Thread-859] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:60620 2023-04-17 03:54:27,796 DEBUG [Thread-859] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:27,797 DEBUG [Thread-855] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:60620 2023-04-17 03:54:27,798 DEBUG [Thread-855] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:27,798 DEBUG [Thread-857] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:60620 2023-04-17 03:54:27,799 DEBUG [Thread-857] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:27,802 DEBUG [Thread-861] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x092bed09 to 127.0.0.1:60620 2023-04-17 03:54:27,802 DEBUG [Thread-861] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:27,812 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ac2dffc46aff4b6c99a5bdbe35cc99b0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ac2dffc46aff4b6c99a5bdbe35cc99b0 2023-04-17 03:54:27,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703727815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703727816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:27,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703727817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703728118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703728118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703728119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,217 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=459, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/1381d1f5154b430fa545af1ecbd23ea3 2023-04-17 03:54:28,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36218 deadline: 1681703728621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36282 deadline: 1681703728622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:28,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36298 deadline: 1681703728623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:28,627 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/8c55b32ffd644d09baf77c16a344d185 2023-04-17 03:54:28,628 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/c0e5fc55a11c493180c87ee0c99c0db1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/c0e5fc55a11c493180c87ee0c99c0db1 2023-04-17 03:54:28,632 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into c0e5fc55a11c493180c87ee0c99c0db1(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 03:54:28,632 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:28,632 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=11, startTime=1681703667342; duration=1sec 2023-04-17 03:54:28,632 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:29,037 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/798945165c874a5fb6f9711f824af424 2023-04-17 03:54:29,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/1381d1f5154b430fa545af1ecbd23ea3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3 2023-04-17 03:54:29,044 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3, entries=150, sequenceid=459, filesize=25.4 K 2023-04-17 03:54:29,045 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/8c55b32ffd644d09baf77c16a344d185 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185 2023-04-17 03:54:29,048 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185, entries=150, sequenceid=459, filesize=11.9 K 2023-04-17 03:54:29,049 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/798945165c874a5fb6f9711f824af424 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424 2023-04-17 03:54:29,052 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424, entries=150, sequenceid=459, filesize=11.9 K 2023-04-17 03:54:29,053 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=46.96 KB/48090 for c6ba4e4e990f87093f177b6844dd61b9 in 1692ms, sequenceid=459, compaction requested=false 2023-04-17 03:54:29,053 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:29,053 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3ac2dc[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:29,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:29,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,059 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:29,060 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,060 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:29,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:29,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:29,063 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:29,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:29,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:29,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:29,063 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:29,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:29,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:29,066 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:29,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:29,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:29,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:29,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:29,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:29,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:29,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:29,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:29,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:29,069 DEBUG [Thread-854] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:29,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:29,069 DEBUG [Thread-854] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:29,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:29,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:29,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:29,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:29,169 DEBUG [Thread-854] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:29,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:54:29,627 DEBUG [Thread-852] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:60620 2023-04-17 03:54:29,627 DEBUG [Thread-850] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:60620 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:29,627 DEBUG [Thread-852] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:29,627 DEBUG [Thread-850] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:29,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:29,631 DEBUG [Thread-848] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:60620 2023-04-17 03:54:29,631 DEBUG [Thread-848] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:30,040 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171a249e1c9cfc4181909225d14080646b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171a249e1c9cfc4181909225d14080646b 2023-04-17 03:54:30,445 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=474, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/be9fac0f5cea4e41a48033931b201a56 2023-04-17 03:54:30,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/62efddd7ea7944a79c37646ee1170435 2023-04-17 03:54:31,137 DEBUG [Thread-846] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:60620 2023-04-17 03:54:31,137 DEBUG [Thread-846] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:31,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/f3f9eef70c2c400084b9a21809d168fd 2023-04-17 03:54:31,267 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/be9fac0f5cea4e41a48033931b201a56 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56 2023-04-17 03:54:31,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56, entries=150, sequenceid=474, filesize=25.4 K 2023-04-17 03:54:31,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/62efddd7ea7944a79c37646ee1170435 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435 2023-04-17 03:54:31,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435, entries=150, sequenceid=474, filesize=11.9 K 2023-04-17 03:54:31,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/f3f9eef70c2c400084b9a21809d168fd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd 2023-04-17 03:54:31,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd, entries=150, sequenceid=474, filesize=11.9 K 2023-04-17 03:54:31,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=13.42 KB/13740 for c6ba4e4e990f87093f177b6844dd61b9 in 1652ms, sequenceid=474, compaction requested=true 2023-04-17 03:54:31,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:31,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:31,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:31,278 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:31,278 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:31,278 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:31,279 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:31,279 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/A is initiating minor compaction (all files) 2023-04-17 03:54:31,279 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:31,279 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/A in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:31,280 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/B is initiating minor compaction (all files) 2023-04-17 03:54:31,280 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/c0e5fc55a11c493180c87ee0c99c0db1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=77.2 K 2023-04-17 03:54:31,280 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/B in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:31,280 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3bb9fadc70aa4784aeb3dc1919b8d982, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=36.6 K 2023-04-17 03:54:31,280 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c0e5fc55a11c493180c87ee0c99c0db1, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:31,280 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3bb9fadc70aa4784aeb3dc1919b8d982, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:31,280 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1381d1f5154b430fa545af1ecbd23ea3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681703666881 2023-04-17 03:54:31,280 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8c55b32ffd644d09baf77c16a344d185, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681703666881 2023-04-17 03:54:31,281 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting be9fac0f5cea4e41a48033931b201a56, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681703667507 2023-04-17 03:54:31,281 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 62efddd7ea7944a79c37646ee1170435, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681703667507 2023-04-17 03:54:31,297 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#B#compaction#198 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:31,308 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#A#compaction#199 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:31,712 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/a6df3fc06aac4ca99338e2540de0e737 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a6df3fc06aac4ca99338e2540de0e737 2023-04-17 03:54:31,716 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/B of c6ba4e4e990f87093f177b6844dd61b9 into a6df3fc06aac4ca99338e2540de0e737(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:31,717 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:31,717 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/B, priority=13, startTime=1681703671278; duration=0sec 2023-04-17 03:54:31,717 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:31,717 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:31,718 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:31,718 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): c6ba4e4e990f87093f177b6844dd61b9/C is initiating minor compaction (all files) 2023-04-17 03:54:31,718 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c6ba4e4e990f87093f177b6844dd61b9/C in TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:31,718 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35634699ac7642cab6e122195dc9a348, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp, totalSize=36.6 K 2023-04-17 03:54:31,718 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 35634699ac7642cab6e122195dc9a348, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1681703666790 2023-04-17 03:54:31,719 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 798945165c874a5fb6f9711f824af424, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1681703666881 2023-04-17 03:54:31,719 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f3f9eef70c2c400084b9a21809d168fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681703667507 2023-04-17 03:54:31,725 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): c6ba4e4e990f87093f177b6844dd61b9#C#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:32,132 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/283d2e14639046bf884ff34e59be6212 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/283d2e14639046bf884ff34e59be6212 2023-04-17 03:54:32,136 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/C of c6ba4e4e990f87093f177b6844dd61b9 into 283d2e14639046bf884ff34e59be6212(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:32,136 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:32,136 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/C, priority=13, startTime=1681703671278; duration=0sec 2023-04-17 03:54:32,136 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:32,528 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/a2e3810a6fb04fc58ec0a7aefeb2ab91 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a2e3810a6fb04fc58ec0a7aefeb2ab91 2023-04-17 03:54:32,533 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c6ba4e4e990f87093f177b6844dd61b9/A of c6ba4e4e990f87093f177b6844dd61b9 into a2e3810a6fb04fc58ec0a7aefeb2ab91(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 03:54:32,533 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:32,533 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9., storeName=c6ba4e4e990f87093f177b6844dd61b9/A, priority=13, startTime=1681703671278; duration=1sec 2023-04-17 03:54:32,533 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:36,091 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-17 03:54:36,198 DEBUG [Thread-844] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33a47c05 to 127.0.0.1:60620 2023-04-17 03:54:36,198 DEBUG [Thread-844] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 7 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 5072 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4835 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 2017 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 6050 rows 2023-04-17 03:54:36,198 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 2016 2023-04-17 03:54:36,199 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 6048 rows 2023-04-17 03:54:36,199 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:54:36,199 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0186ae08 to 127.0.0.1:60620 2023-04-17 03:54:36,199 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:54:36,200 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:54:36,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:54:36,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:36,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:36,205 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703676205"}]},"ts":"1681703676205"} 2023-04-17 03:54:36,206 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:54:36,208 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:54:36,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, UNASSIGN}] 2023-04-17 03:54:36,210 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, UNASSIGN 2023-04-17 03:54:36,210 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:36,210 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703676210"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703676210"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703676210"}]},"ts":"1681703676210"} 2023-04-17 03:54:36,211 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:36,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:36,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c6ba4e4e990f87093f177b6844dd61b9, disabling compactions & flushes 2023-04-17 03:54:36,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. after waiting 0 ms 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:36,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c6ba4e4e990f87093f177b6844dd61b9 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=A 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=B 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c6ba4e4e990f87093f177b6844dd61b9, store=C 2023-04-17 03:54:36,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:36,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:36,775 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417da4f0df86bf54f4eaab0ef78139a454c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da4f0df86bf54f4eaab0ef78139a454c 2023-04-17 03:54:36,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:37,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=483, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/a37e87e77a354f0c93dffac3c5ec13b1 2023-04-17 03:54:37,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:37,589 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/d2740b71520c4628bfc055b8cbe49621 2023-04-17 03:54:37,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=483 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/4f31205ca8bd4280b26a9a5aedd89734 2023-04-17 03:54:37,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/A/a37e87e77a354f0c93dffac3c5ec13b1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a37e87e77a354f0c93dffac3c5ec13b1 2023-04-17 03:54:37,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a37e87e77a354f0c93dffac3c5ec13b1, entries=100, sequenceid=483, filesize=18.6 K 2023-04-17 03:54:37,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/B/d2740b71520c4628bfc055b8cbe49621 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d2740b71520c4628bfc055b8cbe49621 2023-04-17 03:54:37,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d2740b71520c4628bfc055b8cbe49621, entries=100, sequenceid=483, filesize=9.5 K 2023-04-17 03:54:37,611 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/.tmp/C/4f31205ca8bd4280b26a9a5aedd89734 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/4f31205ca8bd4280b26a9a5aedd89734 2023-04-17 03:54:37,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/4f31205ca8bd4280b26a9a5aedd89734, entries=100, sequenceid=483, filesize=9.5 K 2023-04-17 03:54:37,615 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for c6ba4e4e990f87093f177b6844dd61b9 in 1252ms, sequenceid=483, compaction requested=false 2023-04-17 03:54:37,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/47890f81cc3047f4b954213fc254b11d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a1ec00a9d5614c81979542ba9b52ff35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bf1c1ff5792a4dd89179734f93f1f503, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/164cfd2380b84c89abaf168274dec478, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f2fa264710a948ce983f90d5012396a5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/e94a3c1472844fabb6e45a35d96ce8ab, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/c0e5fc55a11c493180c87ee0c99c0db1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56] to archive 2023-04-17 03:54:37,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:37,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d79833c70a2f4875a59eac6f772c7341 2023-04-17 03:54:37,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/0f13d15ef7fc4919bbfe6fa186d72163 2023-04-17 03:54:37,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ffbab29f20b948feaa611c94c78253a7 2023-04-17 03:54:37,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7b5a0e9d52354eeab3c6a754d15b8f7f 2023-04-17 03:54:37,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/29986fd0cec7497aa67c71e25f2d35de 2023-04-17 03:54:37,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/47890f81cc3047f4b954213fc254b11d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/47890f81cc3047f4b954213fc254b11d 2023-04-17 03:54:37,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bd1bccd26b1a49499ceae3fa33c420eb 2023-04-17 03:54:37,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bff38602f1124c9b95232e7340628bef 2023-04-17 03:54:37,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/189bae0eff0d49c89c78e91df9c85af5 2023-04-17 03:54:37,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a1ec00a9d5614c81979542ba9b52ff35 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a1ec00a9d5614c81979542ba9b52ff35 2023-04-17 03:54:37,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bb95cf41b4924bc6a4fb5c62758e3fc2 2023-04-17 03:54:37,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/86ad7d1fab21497fb6a11bbc4763c76f 2023-04-17 03:54:37,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bf1c1ff5792a4dd89179734f93f1f503 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/bf1c1ff5792a4dd89179734f93f1f503 2023-04-17 03:54:37,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/03b543198c214dabad2a313ddb68729d 2023-04-17 03:54:37,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f624f839365141e892df5bbbf2e72cb6 2023-04-17 03:54:37,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/7d11a348f0944770a5c84be02cda5eba 2023-04-17 03:54:37,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/164cfd2380b84c89abaf168274dec478 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/164cfd2380b84c89abaf168274dec478 2023-04-17 03:54:37,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/19a36375df4c4a178d56e7ae65702468 2023-04-17 03:54:37,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/16e6b6b8c567490fb82e672f8bbc22da 2023-04-17 03:54:37,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f2fa264710a948ce983f90d5012396a5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/f2fa264710a948ce983f90d5012396a5 2023-04-17 03:54:37,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/d044383f16a14d9b9901347d69150a6d 2023-04-17 03:54:37,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/8e5f19c2ca7941e0986f70feea8cf8d5 2023-04-17 03:54:37,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/e94a3c1472844fabb6e45a35d96ce8ab to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/e94a3c1472844fabb6e45a35d96ce8ab 2023-04-17 03:54:37,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/655f3e58c18a45a88858be0ad0b1d827 2023-04-17 03:54:37,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1cdc73c4ccc94fb8ba9c893808b7b839 2023-04-17 03:54:37,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/4f582d17805040638ab9c1772c7d6c1c 2023-04-17 03:54:37,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/ec46a0ea29554ac8b46060eca7aad54e 2023-04-17 03:54:37,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/c0e5fc55a11c493180c87ee0c99c0db1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/c0e5fc55a11c493180c87ee0c99c0db1 2023-04-17 03:54:37,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/1381d1f5154b430fa545af1ecbd23ea3 2023-04-17 03:54:37,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/be9fac0f5cea4e41a48033931b201a56 2023-04-17 03:54:37,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/dc0ef2ca8eee44fba438ae924dc351c1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/70c70c16533a4a5682d5cef7aa226302, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b5576e3db0694316a66a712dde342798, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b39439a3d4044a09ae34d3f5e118468b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b7fbb0c172411ea0c90017f2a1f77d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/842e1b38111144c2b9e74f5b4ad34970, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3bb9fadc70aa4784aeb3dc1919b8d982, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435] to archive 2023-04-17 03:54:37,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e6043b43fa144e3a90f74f3f0376f332 2023-04-17 03:54:37,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d53f4ced92294d6c8b57c3277538822f 2023-04-17 03:54:37,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/636aae7f6f9943089f63986c709c2d22 2023-04-17 03:54:37,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e2914e8a572c4fcda635fcafa5c8dd7e 2023-04-17 03:54:37,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/dc0ef2ca8eee44fba438ae924dc351c1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/dc0ef2ca8eee44fba438ae924dc351c1 2023-04-17 03:54:37,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/39d3987c84b14909a4b9ee1a68e2941f 2023-04-17 03:54:37,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/050df742015f41c9bba46a1203537d7d 2023-04-17 03:54:37,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3e65b574e9a045f7a480b3d0b317b0d3 2023-04-17 03:54:37,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/70c70c16533a4a5682d5cef7aa226302 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/70c70c16533a4a5682d5cef7aa226302 2023-04-17 03:54:37,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e26a5d71466a4dde903aeb65046b0f61 2023-04-17 03:54:37,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/2c077c1ae6e74e89a7ee5119c9e999a7 2023-04-17 03:54:37,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b5576e3db0694316a66a712dde342798 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b5576e3db0694316a66a712dde342798 2023-04-17 03:54:37,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/178554f4ce014878ae9dbbacbe98cc6b 2023-04-17 03:54:37,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/566a98d20d0048c7b4824d5c468ec2d6 2023-04-17 03:54:37,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b39439a3d4044a09ae34d3f5e118468b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/b39439a3d4044a09ae34d3f5e118468b 2023-04-17 03:54:37,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/91ba9494b7ed4917889c8413d0bfdc75 2023-04-17 03:54:37,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/9b543224cc374a8ba434a0569260cb02 2023-04-17 03:54:37,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e66176420b0246b581863d163cfa4ab5 2023-04-17 03:54:37,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b7fbb0c172411ea0c90017f2a1f77d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b7fbb0c172411ea0c90017f2a1f77d 2023-04-17 03:54:37,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/264d41d7185a4926a376976e6d511545 2023-04-17 03:54:37,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a96b3e0047ed4df19674ae470b0638da 2023-04-17 03:54:37,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/842e1b38111144c2b9e74f5b4ad34970 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/842e1b38111144c2b9e74f5b4ad34970 2023-04-17 03:54:37,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/7c70daa50d1f4ea4a51a02c8365e6ccc 2023-04-17 03:54:37,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/e9b42a5184ea4cf488ec92d14a7efbe3 2023-04-17 03:54:37,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/4efe196639eb49abacf12403d52b077c 2023-04-17 03:54:37,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/fe8f6a823d9344f580b9d7bb2cab53c9 2023-04-17 03:54:37,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/afa2001aad91409e8989060cca154a7e 2023-04-17 03:54:37,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3bb9fadc70aa4784aeb3dc1919b8d982 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/3bb9fadc70aa4784aeb3dc1919b8d982 2023-04-17 03:54:37,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/8c55b32ffd644d09baf77c16a344d185 2023-04-17 03:54:37,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/62efddd7ea7944a79c37646ee1170435 2023-04-17 03:54:37,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fbc88d28b608475fb984bf5f42222365, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aab4fe09bca14afbbe679932b3fd68e0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/7437dedf4a0a4d47a3e92d0eff01c3e8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/277e4c3b724444d2bdbfc6b6e924fbd6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/034e0bace0054231b8779e0e10ca15f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/1d43304307fd4d51aa926e5701b20cdb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35634699ac7642cab6e122195dc9a348, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd] to archive 2023-04-17 03:54:37,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:54:37,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/37ede0103c394a7da6d7083d64323199 2023-04-17 03:54:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35e05702e96140aeb9aa2c0e16d15bb6 2023-04-17 03:54:37,712 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/67e1615ac27e446ab98c3570d0febe28 2023-04-17 03:54:37,713 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/120e9235a63d44f0b090790180ba08fd 2023-04-17 03:54:37,714 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fbc88d28b608475fb984bf5f42222365 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fbc88d28b608475fb984bf5f42222365 2023-04-17 03:54:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/abf8e36cc4474149b4b68f98207023fb 2023-04-17 03:54:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/b900fb357f0144dfb463b99c85378a8c 2023-04-17 03:54:37,716 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6f8663bea77f495fa692725789c740e4 2023-04-17 03:54:37,717 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aab4fe09bca14afbbe679932b3fd68e0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aab4fe09bca14afbbe679932b3fd68e0 2023-04-17 03:54:37,718 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/389ac38f056149c18b4f6094e76d145f 2023-04-17 03:54:37,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5b1024ec8fe2438585ab477927af0ccd 2023-04-17 03:54:37,719 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/7437dedf4a0a4d47a3e92d0eff01c3e8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/7437dedf4a0a4d47a3e92d0eff01c3e8 2023-04-17 03:54:37,720 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/ffae4322566e46e2bb3d88865b8bfc01 2023-04-17 03:54:37,721 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e80188a83075436a83e1af4713d934dd 2023-04-17 03:54:37,722 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/827c60ac1a2d44818b0842865e6c2bd8 2023-04-17 03:54:37,723 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/277e4c3b724444d2bdbfc6b6e924fbd6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/277e4c3b724444d2bdbfc6b6e924fbd6 2023-04-17 03:54:37,724 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/91ab911cdc1b42569d5b4745b521aab1 2023-04-17 03:54:37,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/10f411d76937440ca3de668a4b160a71 2023-04-17 03:54:37,725 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/034e0bace0054231b8779e0e10ca15f0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/034e0bace0054231b8779e0e10ca15f0 2023-04-17 03:54:37,726 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/fb86186efab0426daae72c10651db5a7 2023-04-17 03:54:37,727 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/0862a28b2b5940ba90e4e489892bf39e 2023-04-17 03:54:37,728 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/1d43304307fd4d51aa926e5701b20cdb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/1d43304307fd4d51aa926e5701b20cdb 2023-04-17 03:54:37,729 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/aac2140e7c144b0c8fd174bd87ab147f 2023-04-17 03:54:37,730 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/5641175241f74a45a04ae867006c7362 2023-04-17 03:54:37,731 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/8a7acb9685af48d894cdf66468dc8f72 2023-04-17 03:54:37,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/e934103aece74c11a2ca26c5ab1281cb 2023-04-17 03:54:37,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35634699ac7642cab6e122195dc9a348 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/35634699ac7642cab6e122195dc9a348 2023-04-17 03:54:37,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/6a6b46ee0eae4705b457d20cccff00d5 2023-04-17 03:54:37,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/798945165c874a5fb6f9711f824af424 2023-04-17 03:54:37,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/f3f9eef70c2c400084b9a21809d168fd 2023-04-17 03:54:37,739 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits/486.seqid, newMaxSeqId=486, maxSeqId=4 2023-04-17 03:54:37,741 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9. 2023-04-17 03:54:37,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c6ba4e4e990f87093f177b6844dd61b9: 2023-04-17 03:54:37,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:37,743 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c6ba4e4e990f87093f177b6844dd61b9, regionState=CLOSED 2023-04-17 03:54:37,743 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703677743"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703677743"}]},"ts":"1681703677743"} 2023-04-17 03:54:37,745 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-17 03:54:37,745 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure c6ba4e4e990f87093f177b6844dd61b9, server=jenkins-hbase4.apache.org,36381,1681703614253 in 1.5330 sec 2023-04-17 03:54:37,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-17 03:54:37,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c6ba4e4e990f87093f177b6844dd61b9, UNASSIGN in 1.5370 sec 2023-04-17 03:54:37,747 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703677747"}]},"ts":"1681703677747"} 2023-04-17 03:54:37,747 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:54:37,750 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:54:37,751 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5490 sec 2023-04-17 03:54:38,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-17 03:54:38,308 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-17 03:54:38,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:54:38,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,311 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 03:54:38,312 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,316 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:38,318 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits] 2023-04-17 03:54:38,321 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a2e3810a6fb04fc58ec0a7aefeb2ab91 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a2e3810a6fb04fc58ec0a7aefeb2ab91 2023-04-17 03:54:38,322 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a37e87e77a354f0c93dffac3c5ec13b1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/A/a37e87e77a354f0c93dffac3c5ec13b1 2023-04-17 03:54:38,324 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a6df3fc06aac4ca99338e2540de0e737 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/a6df3fc06aac4ca99338e2540de0e737 2023-04-17 03:54:38,325 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d2740b71520c4628bfc055b8cbe49621 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/B/d2740b71520c4628bfc055b8cbe49621 2023-04-17 03:54:38,327 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/283d2e14639046bf884ff34e59be6212 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/283d2e14639046bf884ff34e59be6212 2023-04-17 03:54:38,328 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/4f31205ca8bd4280b26a9a5aedd89734 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/C/4f31205ca8bd4280b26a9a5aedd89734 2023-04-17 03:54:38,331 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits/486.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9/recovered.edits/486.seqid 2023-04-17 03:54:38,331 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/c6ba4e4e990f87093f177b6844dd61b9 2023-04-17 03:54:38,331 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:54:38,332 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:54:38,333 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 03:54:38,336 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417002750685b4a409eb2947516260d13cc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417002750685b4a409eb2947516260d13cc 2023-04-17 03:54:38,337 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041700b540637baf4545a33baafcb62abbe4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041700b540637baf4545a33baafcb62abbe4 2023-04-17 03:54:38,338 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417027b59af4a644ac59bb90f9579f0b697 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417027b59af4a644ac59bb90f9579f0b697 2023-04-17 03:54:38,339 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417034df6c14b0c49e9850059c4a6f06939 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417034df6c14b0c49e9850059c4a6f06939 2023-04-17 03:54:38,340 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171a249e1c9cfc4181909225d14080646b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171a249e1c9cfc4181909225d14080646b 2023-04-17 03:54:38,341 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171fb4b708bc8c4f6e90333dc37e39c37d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171fb4b708bc8c4f6e90333dc37e39c37d 2023-04-17 03:54:38,342 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041743fa1276cc964fc7be6e47566ad5865a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041743fa1276cc964fc7be6e47566ad5865a 2023-04-17 03:54:38,343 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174c137d1c15894967a3e17623f6eb78b9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174c137d1c15894967a3e17623f6eb78b9 2023-04-17 03:54:38,344 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041750deb30863d046c8ad64c6421f0499df to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041750deb30863d046c8ad64c6421f0499df 2023-04-17 03:54:38,345 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041761597040f3f2488c8f12db33ffca56f5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041761597040f3f2488c8f12db33ffca56f5 2023-04-17 03:54:38,346 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417679c6341698a4d33bb0283a3869299a9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417679c6341698a4d33bb0283a3869299a9 2023-04-17 03:54:38,347 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417698d00ac1d234f4ea853859a53964981 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417698d00ac1d234f4ea853859a53964981 2023-04-17 03:54:38,347 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177c6922746edf4b4ab1b94e3ba4a03cb0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177c6922746edf4b4ab1b94e3ba4a03cb0 2023-04-17 03:54:38,348 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041785c46a1b111246a5bda5d21f6236ba0d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041785c46a1b111246a5bda5d21f6236ba0d 2023-04-17 03:54:38,349 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178d9db80e869c4061b9e0224dc8fee6c2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304178d9db80e869c4061b9e0224dc8fee6c2 2023-04-17 03:54:38,350 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796657aefc7c447a58a79faf25feaf273 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041796657aefc7c447a58a79faf25feaf273 2023-04-17 03:54:38,351 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a605af031e714312b73d24a5e1fb6515 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a605af031e714312b73d24a5e1fb6515 2023-04-17 03:54:38,352 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ac2dffc46aff4b6c99a5bdbe35cc99b0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ac2dffc46aff4b6c99a5bdbe35cc99b0 2023-04-17 03:54:38,353 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ade643ffe3134fbf903520c5d09d7a10 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ade643ffe3134fbf903520c5d09d7a10 2023-04-17 03:54:38,354 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bcac7fe87d0345c6a00eb0047afec270 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417bcac7fe87d0345c6a00eb0047afec270 2023-04-17 03:54:38,355 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c4471df3a6a34358b6b3c00f8599dc34 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417c4471df3a6a34358b6b3c00f8599dc34 2023-04-17 03:54:38,355 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da4f0df86bf54f4eaab0ef78139a454c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417da4f0df86bf54f4eaab0ef78139a454c 2023-04-17 03:54:38,356 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eb5e09246a7345b2b6ee545f8ead78a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eb5e09246a7345b2b6ee545f8ead78a4 2023-04-17 03:54:38,357 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f188e4febffb45d9a77a4b324b22de9f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f188e4febffb45d9a77a4b324b22de9f 2023-04-17 03:54:38,358 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:54:38,360 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,361 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:54:38,363 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:54:38,364 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,364 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:54:38,364 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703678364"}]},"ts":"9223372036854775807"} 2023-04-17 03:54:38,365 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:54:38,365 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c6ba4e4e990f87093f177b6844dd61b9, NAME => 'TestAcidGuarantees,,1681703644639.c6ba4e4e990f87093f177b6844dd61b9.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:54:38,365 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:54:38,366 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703678366"}]},"ts":"9223372036854775807"} 2023-04-17 03:54:38,367 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:54:38,368 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,369 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 59 msec 2023-04-17 03:54:38,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-17 03:54:38,413 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-17 03:54:38,425 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 235), OpenFileDescriptor=525 (was 566), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=312 (was 390), ProcessCount=169 (was 169), AvailableMemoryMB=686 (was 811) 2023-04-17 03:54:38,436 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=525, MaxFileDescriptor=60000, SystemLoadAverage=312, ProcessCount=169, AvailableMemoryMB=685 2023-04-17 03:54:38,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:54:38,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:54:38,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:54:38,440 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:54:38,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-17 03:54:38,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:38,441 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:54:38,442 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:38,442 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b empty. 2023-04-17 03:54:38,443 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:38,443 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:54:38,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:38,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:38,850 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:54:38,852 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1bb2cde891fcf5bb7edaa5296fcaff7b, NAME => 'TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:54:39,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:39,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:54:39,257 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 1bb2cde891fcf5bb7edaa5296fcaff7b, disabling compactions & flushes 2023-04-17 03:54:39,258 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,258 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,258 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. after waiting 0 ms 2023-04-17 03:54:39,258 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,258 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,258 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:39,260 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:54:39,261 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703679260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703679260"}]},"ts":"1681703679260"} 2023-04-17 03:54:39,262 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:54:39,262 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:54:39,262 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703679262"}]},"ts":"1681703679262"} 2023-04-17 03:54:39,263 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:54:39,266 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, ASSIGN}] 2023-04-17 03:54:39,267 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, ASSIGN 2023-04-17 03:54:39,268 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:54:39,419 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=1bb2cde891fcf5bb7edaa5296fcaff7b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:39,419 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703679419"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703679419"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703679419"}]},"ts":"1681703679419"} 2023-04-17 03:54:39,420 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:54:39,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:39,574 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1bb2cde891fcf5bb7edaa5296fcaff7b, NAME => 'TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:54:39,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:54:39,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,575 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,576 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,578 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A 2023-04-17 03:54:39,578 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A 2023-04-17 03:54:39,578 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:39,578 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bb2cde891fcf5bb7edaa5296fcaff7b columnFamilyName A 2023-04-17 03:54:39,579 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(310): Store=1bb2cde891fcf5bb7edaa5296fcaff7b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:39,579 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,580 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B 2023-04-17 03:54:39,580 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B 2023-04-17 03:54:39,580 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:39,581 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bb2cde891fcf5bb7edaa5296fcaff7b columnFamilyName B 2023-04-17 03:54:39,581 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(310): Store=1bb2cde891fcf5bb7edaa5296fcaff7b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:39,581 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,582 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C 2023-04-17 03:54:39,582 DEBUG [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C 2023-04-17 03:54:39,582 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:54:39,582 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1bb2cde891fcf5bb7edaa5296fcaff7b columnFamilyName C 2023-04-17 03:54:39,583 INFO [StoreOpener-1bb2cde891fcf5bb7edaa5296fcaff7b-1] regionserver.HStore(310): Store=1bb2cde891fcf5bb7edaa5296fcaff7b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:54:39,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:54:39,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:39,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:54:39,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1bb2cde891fcf5bb7edaa5296fcaff7b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69120468, jitterRate=0.02997523546218872}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:54:39,588 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:39,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., pid=30, masterSystemTime=1681703679571 2023-04-17 03:54:39,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:39,590 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=1bb2cde891fcf5bb7edaa5296fcaff7b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:39,590 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703679590"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703679590"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703679590"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703679590"}]},"ts":"1681703679590"} 2023-04-17 03:54:39,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-17 03:54:39,593 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 171 msec 2023-04-17 03:54:39,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-17 03:54:39,594 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, ASSIGN in 327 msec 2023-04-17 03:54:39,594 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:54:39,595 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703679594"}]},"ts":"1681703679594"} 2023-04-17 03:54:39,595 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:54:39,598 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:54:39,599 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1600 sec 2023-04-17 03:54:40,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-17 03:54:40,545 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-17 03:54:40,545 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,551 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,552 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,553 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35402, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,554 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:54:40,556 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:54:40,557 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,562 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,562 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,565 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,565 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,569 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,569 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,572 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,573 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,576 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,576 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,581 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,581 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,587 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,587 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,591 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,591 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,595 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,595 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6f8ff99f to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:54:40,598 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d2b992d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:54:40,603 DEBUG [hconnection-0x153cf553-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,604 DEBUG [hconnection-0x3a56e4f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,604 DEBUG [hconnection-0x77225170-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:40,606 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,606 DEBUG [hconnection-0x4fb121bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,606 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,606 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,607 DEBUG [hconnection-0x3eaf68fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,607 DEBUG [hconnection-0x720da312-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,608 DEBUG [hconnection-0x30a3284-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,608 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35474, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,609 DEBUG [hconnection-0x774ab81-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,610 DEBUG [hconnection-0x6ea852cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,610 DEBUG [hconnection-0x3469fe8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:54:40,611 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,613 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,613 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,617 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,620 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,620 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:54:40,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:40,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:40,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:40,630 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:40,630 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:40,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:40,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:40,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:40,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:40,635 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,635 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:40,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:40,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:40,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,640 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:40,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:40,642 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:40,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:40,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:40,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,651 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd6a42f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,654 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,654 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:40,654 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:40,654 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:40,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:40,655 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,655 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:40,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:40,658 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:40,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:40,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:40,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:40,658 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:40,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:40,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:40,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:40,664 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,664 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:40,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,665 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:40,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703740650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703740681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703740681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703740681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703740682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,765 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:40,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:40,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:40,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:40,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:40,773 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:40,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:40,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:40,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,778 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:40,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:40,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:40,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:40,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:40,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:40,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:40,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703740783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703740784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703740784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703740785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703740785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:40,792 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,792 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:40,792 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:40,792 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:40,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,794 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:40,796 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43c454f1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:40,796 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,797 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,797 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:40,797 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:40,797 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:40,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:40,800 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,800 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:40,800 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,800 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:40,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:40,803 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:40,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:40,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:40,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:40,803 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:40,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,806 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,807 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:40,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:40,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:40,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,810 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,810 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:40,810 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:40,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,911 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:40,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:40,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:40,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:40,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:40,914 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:40,914 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,914 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:40,914 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:40,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,916 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:40,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:40,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:40,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:40,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:40,917 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,917 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:40,917 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:40,918 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:40,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:40,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:40,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:40,918 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:40,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:40,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9aebc91[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:40,922 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:40,922 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,923 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:40,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:40,923 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:40,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,924 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:40,924 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,925 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:40,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:40,927 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:40,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:40,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:40,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:40,927 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:40,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:40,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:40,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:40,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:40,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:40,933 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:40,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:40,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:40,934 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:40,934 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:40,934 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:40,934 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:40,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703740985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703740987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703740988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703740988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:40,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:40,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703740988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,034 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:41,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:41,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:41,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:41,038 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:41,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:41,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:41,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,039 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,039 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,040 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:41,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:41,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:41,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:41,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:41,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:41,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:41,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:41,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:41,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:41,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:41,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:41,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:41,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:41,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20abb35a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:41,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:41,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,046 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:41,046 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:41,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:41,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:41,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:41,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:41,056 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:41,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:41,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:41,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:41,056 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:41,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/f9d12bdae4814467b6d6d1ec97d27766 2023-04-17 03:54:41,060 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:41,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:41,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,064 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:41,064 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,164 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:41,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:41,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:41,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:41,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:41,168 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:41,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:41,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:41,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,169 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,170 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:41,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:41,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:41,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:41,172 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,172 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:41,172 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:41,172 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:41,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:41,174 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,174 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,175 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:41,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78049cbc[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:41,176 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:41,176 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,180 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:41,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:41,180 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:41,180 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,180 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:41,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:41,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:41,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:41,184 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:41,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:41,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:41,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:41,184 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:41,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,186 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,187 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:41,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:41,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:41,190 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,191 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:41,191 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:41,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703741289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703741289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,291 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:41,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:41,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703741292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703741292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703741292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:41,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:41,294 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,296 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:41,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:41,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:41,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:41,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:41,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:41,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:41,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,300 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,300 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,300 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:41,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:41,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19862c91[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:41,302 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:41,302 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:41,303 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:41,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:41,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,305 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:41,305 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,305 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:41,308 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:41,308 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:41,308 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:41,308 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:41,308 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:41,309 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,309 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,310 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,311 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:41,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:41,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,314 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:41,314 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:41,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,414 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:41,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:41,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:41,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:41,418 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:41,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:41,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:41,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,419 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,420 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:41,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:41,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:41,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:41,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:41,421 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:41,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:41,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:41,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65b04104[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:41,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:41,427 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:41,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:41,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:41,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:41,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:41,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:41,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:41,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:41,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:41,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:41,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:41,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:41,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,439 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:41,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:41,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:41,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:41,440 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:41,440 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:41,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:41,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:41,481 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/889b120292094e0ca1f1044949e8caed 2023-04-17 03:54:41,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/63bb94d0057e480282b12b80c7841fc7 2023-04-17 03:54:41,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/f9d12bdae4814467b6d6d1ec97d27766 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766 2023-04-17 03:54:41,502 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 03:54:41,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/889b120292094e0ca1f1044949e8caed as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed 2023-04-17 03:54:41,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 03:54:41,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/63bb94d0057e480282b12b80c7841fc7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7 2023-04-17 03:54:41,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 03:54:41,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 883ms, sequenceid=13, compaction requested=false 2023-04-17 03:54:41,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:41,540 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:41,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:41,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:41,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:41,543 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:41,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:41,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:41,544 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,544 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,545 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:41,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:41,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:41,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:41,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:41,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:41,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:41,547 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:41,547 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:41,547 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:41,547 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:54:41,547 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:41,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:41,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:41,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:41,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:41,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:41,558 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2f22ba2d58ad4d5c97ae00adec9a8f7d 2023-04-17 03:54:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:41,795 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:41,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703741800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703741801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703741801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703741802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703741803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703741904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703741905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703741905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703741905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:41,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703741905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:41,972 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/d190de4bf3cb487e92b7f6baebe9e73c 2023-04-17 03:54:42,004 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/1c93acdfada34a11b1e58ddccd5208e7 2023-04-17 03:54:42,009 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2f22ba2d58ad4d5c97ae00adec9a8f7d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d 2023-04-17 03:54:42,013 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 03:54:42,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/d190de4bf3cb487e92b7f6baebe9e73c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c 2023-04-17 03:54:42,019 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 03:54:42,020 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/1c93acdfada34a11b1e58ddccd5208e7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7 2023-04-17 03:54:42,025 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7, entries=150, sequenceid=33, filesize=11.6 K 2023-04-17 03:54:42,026 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 479ms, sequenceid=33, compaction requested=false 2023-04-17 03:54:42,026 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:42,026 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,032 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18fc0f8b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,036 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,045 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,046 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,046 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,049 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,049 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,050 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,050 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,053 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,057 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,057 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,057 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:42,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:42,108 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:42,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703742120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703742121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703742123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703742123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703742124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,158 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,161 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,164 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,164 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,168 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44418154[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,171 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,171 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,171 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,172 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,174 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,175 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,175 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,175 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,175 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,177 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,181 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,181 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,182 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703742225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703742226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703742226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703742228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703742229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,282 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,285 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,286 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,287 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,287 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,288 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,290 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,290 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c0bdfe9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,299 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,299 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,305 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,305 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,306 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,406 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,409 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,410 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,410 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,411 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,415 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd40b0d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,418 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,420 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,422 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,422 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,424 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703742428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,430 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,431 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,431 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,430 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,431 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,431 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,431 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703742429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703742430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703742430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703742431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/024c8973af924270a387e2e2c85067c6 2023-04-17 03:54:42,531 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/88dc3341cad241d484419c6b2a779446 2023-04-17 03:54:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,536 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,538 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,538 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,538 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,539 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,539 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,539 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,539 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,539 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,540 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,540 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,542 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,542 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732cddab[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,546 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,549 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,549 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,551 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,552 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,555 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,558 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,559 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,558 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,559 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,559 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,659 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,662 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,663 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,663 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,664 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,664 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,665 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,665 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,665 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,665 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,666 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,668 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,668 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bafb9bb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,670 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,672 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,672 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,674 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,674 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,674 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,674 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,674 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,676 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,676 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,676 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,677 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,680 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,680 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,680 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703742729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703742734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703742734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703742734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:42,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703742734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,781 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,784 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,786 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,787 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,787 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,788 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,788 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,789 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,790 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,790 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,791 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@969cdb0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,791 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,792 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,792 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,794 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,794 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,797 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,797 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,797 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,798 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,801 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,805 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,806 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,806 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,906 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:42,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:42,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:42,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:42,910 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:42,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:42,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:42,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,910 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,912 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:42,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:42,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:42,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:42,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:42,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:42,916 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,916 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:42,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@224e33f3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:42,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:42,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:42,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:42,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:42,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,921 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:42,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:42,923 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:42,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:42,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:42,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:42,923 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:42,926 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,926 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,926 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:42,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:42,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:42,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,928 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:42,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:42,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:42,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:42,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:42,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:42,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:42,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:42,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:42,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:42,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:42,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:42,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:42,931 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:42,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:42,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:42,931 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:42,951 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/d17622bf31554a62a5cc5a471cf6c008 2023-04-17 03:54:42,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/024c8973af924270a387e2e2c85067c6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6 2023-04-17 03:54:42,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6, entries=150, sequenceid=46, filesize=11.6 K 2023-04-17 03:54:42,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/88dc3341cad241d484419c6b2a779446 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446 2023-04-17 03:54:42,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446, entries=150, sequenceid=46, filesize=11.6 K 2023-04-17 03:54:42,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/d17622bf31554a62a5cc5a471cf6c008 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008 2023-04-17 03:54:42,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008, entries=150, sequenceid=46, filesize=11.6 K 2023-04-17 03:54:42,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 871ms, sequenceid=46, compaction requested=true 2023-04-17 03:54:42,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:42,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:42,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:42,979 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:42,979 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:42,979 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:42,980 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:42,980 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:42,980 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,980 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=32.5 K 2023-04-17 03:54:42,981 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:42,981 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:42,981 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:42,981 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=32.5 K 2023-04-17 03:54:42,981 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f9d12bdae4814467b6d6d1ec97d27766, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703680625 2023-04-17 03:54:42,982 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 889b120292094e0ca1f1044949e8caed, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703680625 2023-04-17 03:54:42,982 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2f22ba2d58ad4d5c97ae00adec9a8f7d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681703680650 2023-04-17 03:54:42,982 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d190de4bf3cb487e92b7f6baebe9e73c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681703680650 2023-04-17 03:54:42,982 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 024c8973af924270a387e2e2c85067c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:42,983 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 88dc3341cad241d484419c6b2a779446, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:42,990 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#213 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:42,990 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:43,025 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/39493a5516894d02bafcc5d3f4b6c317 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/39493a5516894d02bafcc5d3f4b6c317 2023-04-17 03:54:43,029 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into 39493a5516894d02bafcc5d3f4b6c317(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:43,029 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:43,029 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=13, startTime=1681703682979; duration=0sec 2023-04-17 03:54:43,029 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:43,029 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:43,030 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:43,030 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:43,030 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,030 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=32.5 K 2023-04-17 03:54:43,031 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 63bb94d0057e480282b12b80c7841fc7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703680625 2023-04-17 03:54:43,031 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,031 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1c93acdfada34a11b1e58ddccd5208e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681703680650 2023-04-17 03:54:43,031 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d17622bf31554a62a5cc5a471cf6c008, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:43,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,034 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,039 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,041 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,041 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,041 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:43,042 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,057 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/77e4de9e3464450dbd00e324e4178f07 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/77e4de9e3464450dbd00e324e4178f07 2023-04-17 03:54:43,058 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/8767b35073c0451f8b5e6267d38ce73b 2023-04-17 03:54:43,062 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into 77e4de9e3464450dbd00e324e4178f07(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:43,062 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:43,062 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=13, startTime=1681703682979; duration=0sec 2023-04-17 03:54:43,062 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:43,071 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/e44d0822f849419ab87ebff30114e58a 2023-04-17 03:54:43,080 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8d11116fc8b04ea5ae0cdff4d8be4140 2023-04-17 03:54:43,083 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/8767b35073c0451f8b5e6267d38ce73b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b 2023-04-17 03:54:43,090 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b, entries=150, sequenceid=66, filesize=11.6 K 2023-04-17 03:54:43,091 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/e44d0822f849419ab87ebff30114e58a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a 2023-04-17 03:54:43,097 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a, entries=150, sequenceid=66, filesize=11.6 K 2023-04-17 03:54:43,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8d11116fc8b04ea5ae0cdff4d8be4140 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140 2023-04-17 03:54:43,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140, entries=150, sequenceid=66, filesize=11.6 K 2023-04-17 03:54:43,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 62ms, sequenceid=66, compaction requested=false 2023-04-17 03:54:43,104 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:43,104 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379a7394[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,111 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,114 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,115 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:43,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,121 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,122 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,122 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,222 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,226 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,231 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,231 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,231 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:43,231 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@713e76bd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,235 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,235 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,236 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,237 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,238 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,242 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,243 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,243 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,243 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,243 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:43,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:43,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:43,246 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,246 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,246 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,252 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,252 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e1c3f5cc7f024fbfb885aeb3ca562659 2023-04-17 03:54:43,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703743262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703743264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,268 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:54:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703743265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703743268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703743269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,352 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,355 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,362 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,363 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,363 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,363 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,364 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,364 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,364 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,364 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703743367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,370 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,370 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703743368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36fd2167[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703743370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,373 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,373 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,375 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,375 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703743375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,377 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,378 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,380 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,380 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,380 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703743380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,383 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:43,386 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,386 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,414 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/6940a457ac6b459491ef6da87d5d13ec as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/6940a457ac6b459491ef6da87d5d13ec 2023-04-17 03:54:43,419 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into 6940a457ac6b459491ef6da87d5d13ec(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:43,419 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:43,419 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=13, startTime=1681703682978; duration=0sec 2023-04-17 03:54:43,419 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:43,486 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,489 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,489 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@657f5de0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,496 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,497 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,498 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,500 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,500 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,502 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,502 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,504 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,504 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,508 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,508 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703743569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703743571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703743574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703743578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703743583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,608 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,612 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,613 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,613 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,616 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,617 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,617 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,617 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,617 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,618 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d8408[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,621 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,622 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,622 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,623 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,629 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,633 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,633 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:43,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,679 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/8682783f484944be9df9552424f31d7e 2023-04-17 03:54:43,733 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,736 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,739 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,739 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,740 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,740 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,740 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,740 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,741 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3148118[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,743 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,743 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,750 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,750 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,750 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,750 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,750 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,752 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,752 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,752 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,755 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,757 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,758 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,758 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,758 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,858 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,861 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,861 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,864 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,864 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,866 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,866 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,866 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,866 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,866 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,868 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,869 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c3d7e04[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,870 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,870 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,872 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,872 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703743872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703743873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,875 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,875 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:43,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703743877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703743880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:43,882 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:43,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,883 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,882 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:43,883 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,883 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:43,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:43,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:43,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:43,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703743885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,983 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:43,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:43,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:43,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:43,986 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:43,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:43,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:43,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:43,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:43,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:43,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:43,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:43,990 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,990 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:43,990 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:43,990 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:43,991 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,991 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:43,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:43,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4332b83e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:43,995 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:43,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:43,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:43,995 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,995 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:43,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:43,997 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:43,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:43,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:43,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:43,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:43,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:43,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:44,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:44,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:44,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:44,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:44,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:44,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:44,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:44,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:44,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:44,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:44,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:44,006 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:44,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:44,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:44,006 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:44,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:44,006 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:44,006 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/c39dc7115fa44dc18421f94d9598790d 2023-04-17 03:54:44,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e1c3f5cc7f024fbfb885aeb3ca562659 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659 2023-04-17 03:54:44,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659, entries=250, sequenceid=81, filesize=16.2 K 2023-04-17 03:54:44,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/8682783f484944be9df9552424f31d7e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e 2023-04-17 03:54:44,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e, entries=150, sequenceid=81, filesize=11.6 K 2023-04-17 03:54:44,106 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:44,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/c39dc7115fa44dc18421f94d9598790d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d 2023-04-17 03:54:44,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:44,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:44,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:44,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:44,112 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:44,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:44,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:44,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:44,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:44,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d, entries=150, sequenceid=81, filesize=11.6 K 2023-04-17 03:54:44,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 869ms, sequenceid=81, compaction requested=true 2023-04-17 03:54:44,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:44,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:44,114 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:44,114 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:44,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:44,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:44,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:44,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:44,117 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:44,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:44,117 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:44,117 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:44,117 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/6940a457ac6b459491ef6da87d5d13ec, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=39.5 K 2023-04-17 03:54:44,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6940a457ac6b459491ef6da87d5d13ec, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:44,118 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:44,118 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/39493a5516894d02bafcc5d3f4b6c317, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=34.9 K 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8767b35073c0451f8b5e6267d38ce73b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681703682121 2023-04-17 03:54:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:44,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:44,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 39493a5516894d02bafcc5d3f4b6c317, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:44,118 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e1c3f5cc7f024fbfb885aeb3ca562659, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683234 2023-04-17 03:54:44,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:44,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:44,119 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e44d0822f849419ab87ebff30114e58a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681703682121 2023-04-17 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:44,119 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:44,120 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8682783f484944be9df9552424f31d7e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683240 2023-04-17 03:54:44,120 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:44,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:44,126 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#223 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:44,134 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#224 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:44,145 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/9420ebf28dc244889f1445dafd94ef82 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9420ebf28dc244889f1445dafd94ef82 2023-04-17 03:54:44,148 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/d2e11bc1f8d3495baef497d009346947 2023-04-17 03:54:44,149 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into 9420ebf28dc244889f1445dafd94ef82(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:44,149 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:44,149 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=13, startTime=1681703684113; duration=0sec 2023-04-17 03:54:44,149 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:44,149 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:44,153 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:44,153 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:44,153 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:44,153 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/77e4de9e3464450dbd00e324e4178f07, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=34.9 K 2023-04-17 03:54:44,154 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 77e4de9e3464450dbd00e324e4178f07, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681703681800 2023-04-17 03:54:44,154 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8d11116fc8b04ea5ae0cdff4d8be4140, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681703682121 2023-04-17 03:54:44,155 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c39dc7115fa44dc18421f94d9598790d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683240 2023-04-17 03:54:44,163 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#226 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:44,180 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/cd7898fc37674ef082ca5d6f259aee5f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/cd7898fc37674ef082ca5d6f259aee5f 2023-04-17 03:54:44,184 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into cd7898fc37674ef082ca5d6f259aee5f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:44,185 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:44,185 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=13, startTime=1681703684114; duration=0sec 2023-04-17 03:54:44,185 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:44,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:44,376 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:44,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703744384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703744384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703744385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703744387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703744388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703744487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703744488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703744488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703744490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,548 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/3ba05fab013047aebd3e9f8319f6e8c8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/3ba05fab013047aebd3e9f8319f6e8c8 2023-04-17 03:54:44,552 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into 3ba05fab013047aebd3e9f8319f6e8c8(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:44,552 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:44,552 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=13, startTime=1681703684114; duration=0sec 2023-04-17 03:54:44,552 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:44,566 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/503ed18e1c004e36b747eaaf1f829eae 2023-04-17 03:54:44,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703744689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703744690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703744691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703744692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703744994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703744994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703744995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:44,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:44,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703744996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,004 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/d278db8d6c0048cb9af0474fc10ab016 2023-04-17 03:54:45,009 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/d2e11bc1f8d3495baef497d009346947 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947 2023-04-17 03:54:45,013 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947, entries=150, sequenceid=102, filesize=11.6 K 2023-04-17 03:54:45,014 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/503ed18e1c004e36b747eaaf1f829eae as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae 2023-04-17 03:54:45,019 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae, entries=150, sequenceid=102, filesize=11.6 K 2023-04-17 03:54:45,020 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/d278db8d6c0048cb9af0474fc10ab016 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016 2023-04-17 03:54:45,025 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016, entries=150, sequenceid=102, filesize=11.6 K 2023-04-17 03:54:45,025 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 905ms, sequenceid=102, compaction requested=false 2023-04-17 03:54:45,025 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:45,025 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:45,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:45,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:45,026 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:45,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:45,027 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee2d994[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:45,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:45,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,030 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:45,030 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,031 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:45,038 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:45,039 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:45,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:45,041 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:45,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:45,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:45,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:45,041 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:45,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:45,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:45,044 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:45,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:45,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:45,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:45,048 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:45,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:45,048 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:45,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:45,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:45,149 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:45,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:45,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:45,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:45,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:45,152 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,153 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:45,154 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:45,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:45,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:45,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:45,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:45,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:45,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:45,155 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:45,155 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:45,156 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:45,156 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:45,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:45,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:45,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703745436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703745496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703745496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703745498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703745500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703745538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,566 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2c3e3635de444eb68d92c7d59a3917b1 2023-04-17 03:54:45,576 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/15f3156907cd41e4a4747e471b4d7c92 2023-04-17 03:54:45,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:45,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703745742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:45,986 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/2479e7899aae46c1b9b3fef446c28d85 2023-04-17 03:54:45,990 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2c3e3635de444eb68d92c7d59a3917b1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1 2023-04-17 03:54:45,994 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 03:54:45,995 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/15f3156907cd41e4a4747e471b4d7c92 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92 2023-04-17 03:54:45,998 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 03:54:45,999 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/2479e7899aae46c1b9b3fef446c28d85 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85 2023-04-17 03:54:46,004 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 03:54:46,005 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 848ms, sequenceid=117, compaction requested=true 2023-04-17 03:54:46,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:46,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,009 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9d99111[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,012 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,013 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,016 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,016 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,017 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,017 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,017 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,018 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,022 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,023 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,023 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:46,049 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:46,049 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:46,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703746070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,123 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,126 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,128 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,130 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,130 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,130 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,130 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,131 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,131 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,133 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea31370[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,133 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,134 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,136 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,139 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,139 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,140 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,140 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,140 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,144 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,145 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,145 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703746172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,245 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,248 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,252 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,252 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,252 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,252 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,255 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6806afd2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,257 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,257 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,262 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,262 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,263 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,263 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,264 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,271 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,271 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,371 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,374 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703746376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5090f53f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,383 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,383 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,387 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,387 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,387 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,388 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,388 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,389 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,394 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,396 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,396 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,396 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,396 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,397 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,397 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/079a6f64d1694406b4fd12e93855dc77 2023-04-17 03:54:46,497 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703746498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,500 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703746499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703746507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1584a725[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,510 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,511 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703746510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,514 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,515 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,517 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,522 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,522 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,622 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,627 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,627 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,627 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,627 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,629 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,629 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,629 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,629 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339a1e76[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,636 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,638 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,638 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,640 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,640 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,640 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,642 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,648 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,649 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,649 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:46,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703746679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,749 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,752 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,755 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,755 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,757 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,757 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,757 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,757 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ebf4fe8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,764 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,767 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,767 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,771 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,773 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,773 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/d5fc65b07725419ba3debc30266e7153 2023-04-17 03:54:46,873 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:46,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:46,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:46,877 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:46,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:46,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:46,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,879 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:46,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:46,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:46,881 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:46,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:46,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:46,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d1660d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:46,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:46,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,889 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:46,890 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,890 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:46,890 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:46,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:46,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:46,893 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:46,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:46,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:46,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:46,893 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:46,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:46,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:46,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:46,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:46,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:46,898 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:46,899 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:46,899 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:46,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:46,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:46,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:46,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:46,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:46,999 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:46,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,002 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,005 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,005 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,005 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,005 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:47,007 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,009 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,010 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9dc439d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,012 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,013 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,014 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,016 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,016 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,023 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,023 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,024 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,024 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,124 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,127 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,137 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:47,137 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,139 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad6857e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,142 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,146 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,146 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,148 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,149 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,157 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,157 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:47,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35490 deadline: 1681703747184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,257 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,260 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:47,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e0247c3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,273 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,273 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/6b30a5bd7e5048fa82a6fee58764c7cb 2023-04-17 03:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,284 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,284 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/079a6f64d1694406b4fd12e93855dc77 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77 2023-04-17 03:54:47,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77, entries=200, sequenceid=138, filesize=14.1 K 2023-04-17 03:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/d5fc65b07725419ba3debc30266e7153 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,292 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,292 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,292 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153, entries=150, sequenceid=138, filesize=11.8 K 2023-04-17 03:54:47,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/6b30a5bd7e5048fa82a6fee58764c7cb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb 2023-04-17 03:54:47,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb, entries=150, sequenceid=138, filesize=11.8 K 2023-04-17 03:54:47,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1250ms, sequenceid=138, compaction requested=true 2023-04-17 03:54:47,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:47,299 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:47,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:47,299 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:47,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:47,300 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:47,300 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:47,300 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:47,300 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:47,300 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,301 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,301 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9420ebf28dc244889f1445dafd94ef82, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=49.1 K 2023-04-17 03:54:47,301 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/3ba05fab013047aebd3e9f8319f6e8c8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=46.8 K 2023-04-17 03:54:47,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9420ebf28dc244889f1445dafd94ef82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683240 2023-04-17 03:54:47,301 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3ba05fab013047aebd3e9f8319f6e8c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683240 2023-04-17 03:54:47,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d2e11bc1f8d3495baef497d009346947, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681703683256 2023-04-17 03:54:47,301 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 503ed18e1c004e36b747eaaf1f829eae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681703683256 2023-04-17 03:54:47,302 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2c3e3635de444eb68d92c7d59a3917b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703684378 2023-04-17 03:54:47,302 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 15f3156907cd41e4a4747e471b4d7c92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703684378 2023-04-17 03:54:47,302 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 079a6f64d1694406b4fd12e93855dc77, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685427 2023-04-17 03:54:47,302 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d5fc65b07725419ba3debc30266e7153, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685430 2023-04-17 03:54:47,309 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:47,311 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#235 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:47,321 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/5faad08dba5142f3991ea0b5502061db as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5faad08dba5142f3991ea0b5502061db 2023-04-17 03:54:47,325 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into 5faad08dba5142f3991ea0b5502061db(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:47,325 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,325 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=12, startTime=1681703687299; duration=0sec 2023-04-17 03:54:47,325 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:47,325 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:47,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:47,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:47,327 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,327 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/cd7898fc37674ef082ca5d6f259aee5f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=46.8 K 2023-04-17 03:54:47,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting cd7898fc37674ef082ca5d6f259aee5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1681703683240 2023-04-17 03:54:47,328 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d278db8d6c0048cb9af0474fc10ab016, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1681703683256 2023-04-17 03:54:47,328 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2479e7899aae46c1b9b3fef446c28d85, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703684378 2023-04-17 03:54:47,328 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6b30a5bd7e5048fa82a6fee58764c7cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685430 2023-04-17 03:54:47,335 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:47,343 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/18517c894ace4df5995542e4b47e33f2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/18517c894ace4df5995542e4b47e33f2 2023-04-17 03:54:47,347 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into 18517c894ace4df5995542e4b47e33f2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:47,347 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,347 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=12, startTime=1681703687299; duration=0sec 2023-04-17 03:54:47,347 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:47,393 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,396 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,399 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,400 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:47,400 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:47,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:47,407 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/113d4a1c2529433992cec986baafa299 2023-04-17 03:54:47,417 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/ad6d284580ad4a5d8772ff7e350c2d0f 2023-04-17 03:54:47,427 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=151 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8afcc94faaa84f178318570a4f80e720 2023-04-17 03:54:47,430 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/113d4a1c2529433992cec986baafa299 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299 2023-04-17 03:54:47,434 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299, entries=150, sequenceid=151, filesize=11.8 K 2023-04-17 03:54:47,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/ad6d284580ad4a5d8772ff7e350c2d0f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f 2023-04-17 03:54:47,439 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f, entries=150, sequenceid=151, filesize=11.8 K 2023-04-17 03:54:47,439 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8afcc94faaa84f178318570a4f80e720 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720 2023-04-17 03:54:47,443 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720, entries=150, sequenceid=151, filesize=11.8 K 2023-04-17 03:54:47,443 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 43ms, sequenceid=151, compaction requested=false 2023-04-17 03:54:47,444 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,444 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,446 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,447 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,447 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,447 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd1a700[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,450 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,451 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,454 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,455 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,456 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,456 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,456 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,462 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,462 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,562 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,566 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,568 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,570 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,570 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,570 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,570 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,575 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fac5ac[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,579 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,579 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,582 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,582 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,588 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,588 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,589 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,688 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,692 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,693 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,694 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,694 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d95958[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,702 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,705 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,705 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,707 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,708 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,712 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,712 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,712 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,712 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,712 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,727 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/d839c67fdc75487cadc3a76107ab4251 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d839c67fdc75487cadc3a76107ab4251 2023-04-17 03:54:47,732 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into d839c67fdc75487cadc3a76107ab4251(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:47,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,732 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=12, startTime=1681703687299; duration=0sec 2023-04-17 03:54:47,732 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:47,812 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,817 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,817 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,817 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,817 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,819 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,819 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,819 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,819 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,821 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,821 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,821 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,821 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@158935ff[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,827 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,832 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,832 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,838 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,839 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,839 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,939 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:47,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:47,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:47,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:47,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:47,943 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:47,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:47,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:47,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:47,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:47,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:47,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:47,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:47,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:47,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54a179e9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:47,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:47,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,953 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:47,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:47,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:47,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:47,961 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:47,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:47,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:47,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:47,961 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:47,962 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:47,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:47,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:47,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:47,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:47,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:47,967 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:47,967 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:47,967 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:47,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,067 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,071 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:48,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25071174[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,081 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,083 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,085 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,085 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,094 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,094 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,195 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:48,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:48,198 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:48,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:48,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:48,198 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:48,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,200 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,201 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,204 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,206 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a78f15[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,209 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,209 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,215 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,215 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:48,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,220 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,221 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703688224, max timestamp=1681703688245Num uniques 859; 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703688224, max timestamp=1681703688245Num uniques 859; 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703688224, max timestamp=1681703688245Num uniques 859; 2023-04-17 03:54:48,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703748292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,321 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,324 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,330 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4add66f3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,334 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,342 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,343 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:48,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,346 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,346 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703748395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,446 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,449 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,451 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,453 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,454 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@156c3070[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,458 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,458 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,458 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,462 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,462 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,464 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,464 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,464 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,468 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,468 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,468 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:48,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703748509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,512 DEBUG [Thread-1665] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:48,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703748515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,517 DEBUG [Thread-1663] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:48,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703748521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,525 DEBUG [Thread-1659] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:48,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703748525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,528 DEBUG [Thread-1661] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:48,568 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,571 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,576 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,576 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,576 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,576 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,577 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87d1f18[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,581 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,581 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,581 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,582 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,582 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,583 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,585 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,585 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,587 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,587 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,587 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,592 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,592 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,592 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703748599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/fff44f43f2d94358a8d75c2931b86e26 2023-04-17 03:54:48,692 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,696 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,698 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,699 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,700 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,701 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,703 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2288e625[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,703 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,705 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,706 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,709 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,709 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,714 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,719 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,719 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,820 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,823 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,824 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,824 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,826 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,826 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,826 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,826 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,828 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c3f679[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,831 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,831 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,836 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,836 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,838 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,838 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,838 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,842 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,842 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:48,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:48,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703748903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,942 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:48,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:48,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:48,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:48,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:48,945 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:48,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:48,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:48,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,947 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:48,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:48,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:48,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:48,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:48,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:48,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,952 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:48,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:48,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d8b9c1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:48,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:48,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:48,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:48,957 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:48,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:48,959 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:48,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:48,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:48,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:48,959 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:48,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:48,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:48,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:48,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:48,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:48,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:48,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:48,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:48,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:48,966 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:48,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:48,966 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:49,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/0bc4730ffdb840b58d2579905ccc4cac 2023-04-17 03:54:49,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/9503e206d8f94e6da7da52273084609a 2023-04-17 03:54:49,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/fff44f43f2d94358a8d75c2931b86e26 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26 2023-04-17 03:54:49,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26, entries=150, sequenceid=167, filesize=11.8 K 2023-04-17 03:54:49,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/0bc4730ffdb840b58d2579905ccc4cac as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac 2023-04-17 03:54:49,040 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac, entries=150, sequenceid=167, filesize=11.8 K 2023-04-17 03:54:49,041 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/9503e206d8f94e6da7da52273084609a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a 2023-04-17 03:54:49,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a, entries=150, sequenceid=167, filesize=11.8 K 2023-04-17 03:54:49,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 847ms, sequenceid=167, compaction requested=true 2023-04-17 03:54:49,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:49,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:49,045 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:49,045 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:49,046 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:49,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:49,046 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:49,047 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:49,047 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:49,047 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5faad08dba5142f3991ea0b5502061db, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=35.6 K 2023-04-17 03:54:49,047 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d839c67fdc75487cadc3a76107ab4251, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=35.6 K 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5faad08dba5142f3991ea0b5502061db, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685430 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d839c67fdc75487cadc3a76107ab4251, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685430 2023-04-17 03:54:49,047 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 113d4a1c2529433992cec986baafa299, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681703686061 2023-04-17 03:54:49,048 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ad6d284580ad4a5d8772ff7e350c2d0f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681703686061 2023-04-17 03:54:49,048 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fff44f43f2d94358a8d75c2931b86e26, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:49,048 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0bc4730ffdb840b58d2579905ccc4cac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:49,055 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:49,055 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:49,064 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/9c106cbed95e4efea2c9fcd40d7e3686 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9c106cbed95e4efea2c9fcd40d7e3686 2023-04-17 03:54:49,067 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:49,068 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into 9c106cbed95e4efea2c9fcd40d7e3686(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:49,068 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:49,068 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=13, startTime=1681703689045; duration=0sec 2023-04-17 03:54:49,068 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:49,069 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:54:49,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:49,070 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:54:49,070 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:49,070 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:49,070 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/18517c894ace4df5995542e4b47e33f2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=35.6 K 2023-04-17 03:54:49,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:49,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:49,070 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:49,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:49,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:49,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:49,070 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 18517c894ace4df5995542e4b47e33f2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681703685430 2023-04-17 03:54:49,071 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8afcc94faaa84f178318570a4f80e720, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=151, earliestPutTs=1681703686061 2023-04-17 03:54:49,071 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9503e206d8f94e6da7da52273084609a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:49,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:49,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:49,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:49,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:49,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:49,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:49,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:49,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:49,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:49,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:49,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:49,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:49,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:49,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:49,075 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:49,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:49,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:49,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:49,078 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:49,097 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/bd1d78ab6f7c45eba7ffc3f0516471bd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/bd1d78ab6f7c45eba7ffc3f0516471bd 2023-04-17 03:54:49,101 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into bd1d78ab6f7c45eba7ffc3f0516471bd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:49,101 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:49,101 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=13, startTime=1681703689046; duration=0sec 2023-04-17 03:54:49,101 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:49,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:49,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:49,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:49,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703749424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:49,467 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/653da77fa6b4413b831def56c717e5cb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/653da77fa6b4413b831def56c717e5cb 2023-04-17 03:54:49,471 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into 653da77fa6b4413b831def56c717e5cb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:49,471 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:49,471 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=13, startTime=1681703689045; duration=0sec 2023-04-17 03:54:49,471 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:49,498 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e42b4116d954442496d927a86a593c5d 2023-04-17 03:54:49,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703749526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:49,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703749730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:49,907 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/384879b9714442d78d7d1a479a048ed4 2023-04-17 03:54:50,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:50,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703750034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,318 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/ad7af4fafba44939a8b07d57d2bf3758 2023-04-17 03:54:50,322 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e42b4116d954442496d927a86a593c5d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d 2023-04-17 03:54:50,326 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d, entries=150, sequenceid=205, filesize=11.8 K 2023-04-17 03:54:50,327 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/384879b9714442d78d7d1a479a048ed4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4 2023-04-17 03:54:50,330 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4, entries=150, sequenceid=205, filesize=11.8 K 2023-04-17 03:54:50,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/ad7af4fafba44939a8b07d57d2bf3758 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758 2023-04-17 03:54:50,334 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758, entries=150, sequenceid=205, filesize=11.8 K 2023-04-17 03:54:50,335 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1260ms, sequenceid=205, compaction requested=false 2023-04-17 03:54:50,335 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:50,335 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:50,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17bdedfd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:50,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:50,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:50,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:50,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:50,345 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:50,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:50,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:50,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:50,345 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:50,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:50,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,351 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:50,351 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:50,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:50,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,451 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:50,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:50,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:50,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:50,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:50,454 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:50,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:50,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:50,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:50,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:50,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:50,462 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,462 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:50,462 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:50,462 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:50,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:50,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:50,463 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,470 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/b78d0b4d442143309402dc4ef9414f7f 2023-04-17 03:54:50,480 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/bbbec1ed5c41427b900afb70747f8bf9 2023-04-17 03:54:50,489 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/077bbaa874d8402780a6b4641e8c79fe 2023-04-17 03:54:50,493 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/b78d0b4d442143309402dc4ef9414f7f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f 2023-04-17 03:54:50,497 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f, entries=150, sequenceid=217, filesize=11.8 K 2023-04-17 03:54:50,498 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/bbbec1ed5c41427b900afb70747f8bf9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9 2023-04-17 03:54:50,501 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9, entries=150, sequenceid=217, filesize=11.8 K 2023-04-17 03:54:50,502 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/077bbaa874d8402780a6b4641e8c79fe as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe 2023-04-17 03:54:50,505 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe, entries=150, sequenceid=217, filesize=11.8 K 2023-04-17 03:54:50,506 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 44ms, sequenceid=217, compaction requested=true 2023-04-17 03:54:50,506 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:50,506 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:50,506 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:50,508 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726188e1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:50,512 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,512 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:50,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:50,516 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:50,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:50,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:50,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:50,516 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:50,519 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,519 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,519 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,523 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,523 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,523 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,524 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:50,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,524 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:50,524 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:50,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:50,551 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:50,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/c9cece6777d4447c995fb820258aeafe 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703690586, max timestamp=1681703690609Num uniques 859; 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703690586, max timestamp=1681703690609Num uniques 859; 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703690586, max timestamp=1681703690609Num uniques 859; 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:50,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:50,624 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:50,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:50,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:50,627 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:50,627 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,629 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,629 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:50,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:50,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,631 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:50,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:50,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:50,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:50,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:50,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8dd8ea[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:50,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,647 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:50,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:50,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:50,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:50,648 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:50,651 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:50,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:50,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:50,651 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:50,651 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:50,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,652 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,653 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,653 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,654 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:50,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,657 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:50,657 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:50,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703750671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,757 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:50,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:50,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:50,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:50,760 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:50,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,763 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:50,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:50,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:50,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:50,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:50,765 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:50,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:50,766 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c29724b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:50,768 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:50,768 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:50,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:50,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:50,773 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:50,774 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:50,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:50,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703750775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:50,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:50,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,780 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:50,780 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:50,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,880 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:50,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:50,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:50,884 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,885 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,886 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,886 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:50,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:50,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:50,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:50,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:50,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,893 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:50,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2483be36[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:50,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:50,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,896 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:50,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:50,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:50,897 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:50,900 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:50,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:50,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:50,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:50,900 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:50,901 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,901 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:50,901 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,902 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:50,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:50,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:50,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:50,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:50,906 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:50,907 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:50,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/42ed926b18f8433ba237ab6bb9f601cb 2023-04-17 03:54:50,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703750978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,007 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:51,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:51,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:51,010 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:51,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:51,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:51,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,011 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:51,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:51,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:51,013 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:51,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:51,013 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:51,014 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:51,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:51,015 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a5b0cc7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:51,017 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:51,017 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:51,018 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,018 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:51,025 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:51,028 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:51,028 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:51,028 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:51,028 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:51,028 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:51,029 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,029 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,029 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,030 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,031 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:51,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:51,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,033 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:51,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,034 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:51,034 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,034 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,034 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,134 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:51,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:51,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:51,137 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:51,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:51,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:51,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,138 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,138 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:51,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:51,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:51,140 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,140 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:51,140 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:51,140 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:51,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:51,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:51,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:51,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a11c5ee[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:51,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,146 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:51,146 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:51,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:51,150 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:51,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:51,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:51,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:51,150 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:51,151 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,151 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:51,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:51,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,157 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:51,157 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:51,257 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:51,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:51,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:51,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:51,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:51,261 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:51,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:51,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:51,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,262 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,262 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:51,264 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:51,264 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:51,264 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,264 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:51,265 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:51,265 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:51,265 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:51,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:51,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3d5fcd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:51,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:51,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,269 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:51,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:51,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:51,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:51,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:51,278 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:51,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:51,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:51,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:51,278 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:51,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,280 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703751281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:51,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:51,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,284 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:51,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:51,284 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:51,284 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,376 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8d203f4b7f844770882d0326e80e1a4d 2023-04-17 03:54:51,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/c9cece6777d4447c995fb820258aeafe as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe 2023-04-17 03:54:51,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe, entries=150, sequenceid=227, filesize=11.8 K 2023-04-17 03:54:51,385 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:51,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/42ed926b18f8433ba237ab6bb9f601cb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb 2023-04-17 03:54:51,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:51,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:51,388 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,389 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb, entries=150, sequenceid=227, filesize=11.8 K 2023-04-17 03:54:51,389 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8d203f4b7f844770882d0326e80e1a4d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d 2023-04-17 03:54:51,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:51,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:51,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:51,391 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,391 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:51,391 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:51,391 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:51,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:51,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d, entries=150, sequenceid=227, filesize=11.8 K 2023-04-17 03:54:51,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 843ms, sequenceid=227, compaction requested=true 2023-04-17 03:54:51,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:51,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:51,394 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,394 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:51,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:51,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,396 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:51,396 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,396 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,396 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/653da77fa6b4413b831def56c717e5cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=47.4 K 2023-04-17 03:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,396 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:51,396 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:51,396 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:51,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f63a52[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:51,396 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9c106cbed95e4efea2c9fcd40d7e3686, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=47.4 K 2023-04-17 03:54:51,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:51,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,397 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 653da77fa6b4413b831def56c717e5cb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:51,397 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9c106cbed95e4efea2c9fcd40d7e3686, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:51,397 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e42b4116d954442496d927a86a593c5d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681703688283 2023-04-17 03:54:51,397 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 384879b9714442d78d7d1a479a048ed4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681703688283 2023-04-17 03:54:51,398 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b78d0b4d442143309402dc4ef9414f7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681703689415 2023-04-17 03:54:51,398 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:51,398 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:51,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:51,398 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting bbbec1ed5c41427b900afb70747f8bf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681703689415 2023-04-17 03:54:51,398 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c9cece6777d4447c995fb820258aeafe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:51,399 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 42ed926b18f8433ba237ab6bb9f601cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:51,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:51,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:51,405 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:51,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:51,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:51,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:51,405 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:51,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:51,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:51,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:51,410 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:51,411 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:51,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:51,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:51,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:51,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:51,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:51,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:51,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,416 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:51,416 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:51,516 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:51,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:51,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:51,519 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:51,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:51,519 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:51,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:51,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:51,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:51,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:51,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:51,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:51,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:51,523 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:51,523 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:51,524 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:51,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:51,786 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703751809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,820 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/b3d8c1dcc0884b119ec6009b305df723 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b3d8c1dcc0884b119ec6009b305df723 2023-04-17 03:54:51,821 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/5276e4c1990b487797519f78838da3ea as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/5276e4c1990b487797519f78838da3ea 2023-04-17 03:54:51,825 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into b3d8c1dcc0884b119ec6009b305df723(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:51,825 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:51,825 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=12, startTime=1681703691394; duration=0sec 2023-04-17 03:54:51,825 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:51,825 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:54:51,826 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48588 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:54:51,826 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:51,827 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:51,827 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/bd1d78ab6f7c45eba7ffc3f0516471bd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=47.4 K 2023-04-17 03:54:51,827 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bd1d78ab6f7c45eba7ffc3f0516471bd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1681703688192 2023-04-17 03:54:51,827 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into 5276e4c1990b487797519f78838da3ea(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:51,827 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:51,827 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=12, startTime=1681703691394; duration=0sec 2023-04-17 03:54:51,827 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:51,827 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ad7af4fafba44939a8b07d57d2bf3758, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1681703688283 2023-04-17 03:54:51,828 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 077bbaa874d8402780a6b4641e8c79fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1681703689415 2023-04-17 03:54:51,828 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8d203f4b7f844770882d0326e80e1a4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:51,834 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:51,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703751913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:51,931 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/dced4301c3234b0bbfc9f46d1e1edb00 2023-04-17 03:54:52,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703752115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,242 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/8f474143dadd4feeab7ef14e85d55a54 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8f474143dadd4feeab7ef14e85d55a54 2023-04-17 03:54:52,247 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into 8f474143dadd4feeab7ef14e85d55a54(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:52,247 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:52,247 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=12, startTime=1681703691394; duration=0sec 2023-04-17 03:54:52,247 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:52,342 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/9d43b1e87c53481e81e24ba87cd84dc7 2023-04-17 03:54:52,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703752418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35416 deadline: 1681703752519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,523 DEBUG [Thread-1663] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:52,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35474 deadline: 1681703752532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,534 DEBUG [Thread-1659] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:52,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35432 deadline: 1681703752549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,552 DEBUG [Thread-1665] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:52,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:52,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35458 deadline: 1681703752553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,556 DEBUG [Thread-1661] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:54:52,752 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/f1276b40da68461c9a3f78f028c4532f 2023-04-17 03:54:52,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/dced4301c3234b0bbfc9f46d1e1edb00 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00 2023-04-17 03:54:52,763 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00, entries=150, sequenceid=264, filesize=11.9 K 2023-04-17 03:54:52,763 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/9d43b1e87c53481e81e24ba87cd84dc7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7 2023-04-17 03:54:52,767 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7, entries=150, sequenceid=264, filesize=11.9 K 2023-04-17 03:54:52,768 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/f1276b40da68461c9a3f78f028c4532f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f 2023-04-17 03:54:52,771 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f, entries=150, sequenceid=264, filesize=11.9 K 2023-04-17 03:54:52,772 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1249ms, sequenceid=264, compaction requested=false 2023-04-17 03:54:52,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:52,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:52,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,774 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:52,774 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,774 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76493d3f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:52,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:52,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:52,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,777 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:52,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:52,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:52,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:52,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:52,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:52,782 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:52,782 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:52,782 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:52,782 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:52,782 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:52,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:52,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:52,786 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:52,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:52,789 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:52,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:52,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:52,789 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:52,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:52,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,789 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:52,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:52,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:52,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:52,890 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:52,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:52,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:52,893 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,893 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:52,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:52,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:52,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:52,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:52,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:52,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:52,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:52,896 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:52,897 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:52,897 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:52,904 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/0f5080b4a6594662873e92bd9f4c2e6c 2023-04-17 03:54:52,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:52,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703692940, max timestamp=1681703692961Num uniques 859; 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703692940, max timestamp=1681703692961Num uniques 859; 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703692940, max timestamp=1681703692961Num uniques 859; 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:52,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:53,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703753004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:53,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703753106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:53,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703753309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,317 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/90c1e7a6a2f7401b8b3982fbb3d36c99 2023-04-17 03:54:53,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:53,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703753614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,727 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/a81f1569e31f4c91b037f2140e3b397b 2023-04-17 03:54:53,731 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/0f5080b4a6594662873e92bd9f4c2e6c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c 2023-04-17 03:54:53,735 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c, entries=100, sequenceid=277, filesize=9.5 K 2023-04-17 03:54:53,736 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/90c1e7a6a2f7401b8b3982fbb3d36c99 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99 2023-04-17 03:54:53,739 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99, entries=100, sequenceid=277, filesize=9.5 K 2023-04-17 03:54:53,739 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/a81f1569e31f4c91b037f2140e3b397b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b 2023-04-17 03:54:53,742 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b, entries=100, sequenceid=277, filesize=9.5 K 2023-04-17 03:54:53,743 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 846ms, sequenceid=277, compaction requested=true 2023-04-17 03:54:53,743 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:53,743 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:53,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:53,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:53,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:53,748 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:53,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a3a8ad[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:53,748 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,749 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:53,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:53,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:53,750 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,750 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:53,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:53,753 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:53,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:53,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:53,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:53,753 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:53,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,758 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:53,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:53,759 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:53,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:53,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:53,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:53,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:53,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:53,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:53,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,762 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:53,762 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:53,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:53,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:53,862 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:53,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:53,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:53,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:53,867 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:53,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:53,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:53,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:53,869 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:53,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:53,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:53,870 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:53,870 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:53,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:53,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:53,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:53,871 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:53,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:53,879 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e005fff346d741009d8d845fd63e2ade 2023-04-17 03:54:53,889 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/cec90908c33c43b1b1ae803662f4a038 2023-04-17 03:54:54,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:54,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:54,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703754142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:54,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703754246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,299 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e4dc3b045514428894674e82c818bef2 2023-04-17 03:54:54,303 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/e005fff346d741009d8d845fd63e2ade as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade 2023-04-17 03:54:54,311 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade, entries=150, sequenceid=314, filesize=11.9 K 2023-04-17 03:54:54,312 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/cec90908c33c43b1b1ae803662f4a038 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038 2023-04-17 03:54:54,316 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038, entries=150, sequenceid=314, filesize=11.9 K 2023-04-17 03:54:54,317 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e4dc3b045514428894674e82c818bef2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2 2023-04-17 03:54:54,320 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2, entries=150, sequenceid=314, filesize=11.9 K 2023-04-17 03:54:54,321 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 450ms, sequenceid=314, compaction requested=true 2023-04-17 03:54:54,321 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:54,321 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,323 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5475803[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,337 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,338 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,338 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,340 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,341 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,342 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,343 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:54,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,346 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,346 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,446 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:54,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:54,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:54,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:54,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:54,450 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:54,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:54,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:54,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:54,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:54,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:54,453 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,454 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:54,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,455 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:54,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:54,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:54,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,457 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,457 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,458 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,458 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a6d653[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,461 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,463 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,465 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,466 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,469 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,472 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,472 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703694472, max timestamp=1681703694515Num uniques 859; 2023-04-17 03:54:54,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:54,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703694472, max timestamp=1681703694515Num uniques 859; 2023-04-17 03:54:54,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:54,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703694472, max timestamp=1681703694515Num uniques 859; 2023-04-17 03:54:54,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:54,572 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:54,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:54,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703754570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:54,575 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:54,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:54,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:54,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,577 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:54,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb9abbd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,584 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,585 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,587 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,587 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,587 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,588 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,589 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,589 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,589 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:54,597 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,597 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,597 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:54,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:54,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703754674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,697 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:54,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:54,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:54,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:54,700 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:54,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:54,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:54,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,702 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,703 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:54,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:54,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:54,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:54,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:54,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40608227[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,707 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,707 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,708 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,709 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,711 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,713 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,713 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,713 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,720 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,720 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:54,820 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:54,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:54,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:54,824 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:54,824 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,824 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:54,824 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:54,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,825 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,826 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:54,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:54,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,832 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,833 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd84d68[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,833 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,838 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,839 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,841 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,845 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,846 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,847 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:54,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/192082a2477f4d54b16529895ade0a04 2023-04-17 03:54:54,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:54,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703754878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,947 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:54,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:54,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:54,950 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:54,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:54,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:54,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:54,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:54,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:54,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:54,953 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,953 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:54,953 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:54,953 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:54,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:54,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93a188e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:54,956 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:54,956 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,957 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:54,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:54,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:54,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:54,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:54,963 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:54,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:54,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:54,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:54,963 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:54,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:54,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:54,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:54,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:54,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:54,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:54,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:54,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:54,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:54,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:54,969 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:54,969 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:55,070 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:55,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:55,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:55,073 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:55,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:55,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:55,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:55,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:55,078 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,078 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,078 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:55,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:55,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:55,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:55,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:55,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:55,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:55,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:55,080 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:55,081 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:55,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6cae57[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:55,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:55,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,084 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:55,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:55,084 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,084 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:55,085 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:55,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,086 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:55,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:55,088 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:55,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:55,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:55,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:55,088 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:55,090 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,090 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,090 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:55,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:55,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:55,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:55,095 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:55,095 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:55,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:55,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:55,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:55,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703755182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,195 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:55,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:55,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:55,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:55,198 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:55,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:55,198 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:55,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,200 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:55,200 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:55,200 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,200 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:55,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:55,201 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:55,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:55,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:55,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:55,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:55,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,203 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:55,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:55,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5962a7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:55,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:55,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:55,214 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:55,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:55,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:55,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:55,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:55,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:55,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:55,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:55,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:55,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:55,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:55,222 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:55,222 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:55,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:55,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:55,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:55,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:55,269 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/c54a47b846f548e98c7b137f53406fa7 2023-04-17 03:54:55,280 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e1df866d12f04406b5351fdefb0ef21a 2023-04-17 03:54:55,284 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/192082a2477f4d54b16529895ade0a04 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04 2023-04-17 03:54:55,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04, entries=200, sequenceid=325, filesize=14.3 K 2023-04-17 03:54:55,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/c54a47b846f548e98c7b137f53406fa7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7 2023-04-17 03:54:55,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7, entries=150, sequenceid=325, filesize=11.9 K 2023-04-17 03:54:55,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e1df866d12f04406b5351fdefb0ef21a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a 2023-04-17 03:54:55,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a, entries=150, sequenceid=325, filesize=11.9 K 2023-04-17 03:54:55,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 843ms, sequenceid=325, compaction requested=true 2023-04-17 03:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:55,295 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:55,295 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:55,295 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:55,300 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58916 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:55,300 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/B is initiating minor compaction (all files) 2023-04-17 03:54:55,300 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/B in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,300 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61356 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:55,300 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/5276e4c1990b487797519f78838da3ea, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=57.5 K 2023-04-17 03:54:55,300 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/A is initiating minor compaction (all files) 2023-04-17 03:54:55,301 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/A in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,301 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b3d8c1dcc0884b119ec6009b305df723, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=59.9 K 2023-04-17 03:54:55,301 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5276e4c1990b487797519f78838da3ea, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:55,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b3d8c1dcc0884b119ec6009b305df723, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:55,301 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9d43b1e87c53481e81e24ba87cd84dc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681703690653 2023-04-17 03:54:55,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting dced4301c3234b0bbfc9f46d1e1edb00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681703690653 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 90c1e7a6a2f7401b8b3982fbb3d36c99, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681703691803 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0f5080b4a6594662873e92bd9f4c2e6c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681703691803 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cec90908c33c43b1b1ae803662f4a038, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703692991 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e005fff346d741009d8d845fd63e2ade, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703692991 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c54a47b846f548e98c7b137f53406fa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681703694131 2023-04-17 03:54:55,302 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 192082a2477f4d54b16529895ade0a04, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681703694131 2023-04-17 03:54:55,310 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#A#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:55,311 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#B#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:55,321 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/06defb93105a46ecaa1fcebe8d046200 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/06defb93105a46ecaa1fcebe8d046200 2023-04-17 03:54:55,322 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:55,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:55,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:55,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:55,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:55,325 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:55,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:55,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:55,325 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/B of 1bb2cde891fcf5bb7edaa5296fcaff7b into 06defb93105a46ecaa1fcebe8d046200(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:55,325 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:55,325 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/B, priority=11, startTime=1681703695295; duration=0sec 2023-04-17 03:54:55,326 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:54:55,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,326 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:54:55,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,327 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58916 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:54:55,327 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 1bb2cde891fcf5bb7edaa5296fcaff7b/C is initiating minor compaction (all files) 2023-04-17 03:54:55,327 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 1bb2cde891fcf5bb7edaa5296fcaff7b/C in TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,327 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8f474143dadd4feeab7ef14e85d55a54, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp, totalSize=57.5 K 2023-04-17 03:54:55,328 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8f474143dadd4feeab7ef14e85d55a54, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1681703690544 2023-04-17 03:54:55,328 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f1276b40da68461c9a3f78f028c4532f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1681703690653 2023-04-17 03:54:55,328 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:55,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:55,328 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting a81f1569e31f4c91b037f2140e3b397b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1681703691803 2023-04-17 03:54:55,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,329 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e4dc3b045514428894674e82c818bef2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703692991 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:55,329 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e1df866d12f04406b5351fdefb0ef21a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1681703694131 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:55,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:55,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:55,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:55,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:55,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:55,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:55,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:55,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:55,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:55,330 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 03:54:55,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:55,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:55,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:55,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:55,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:55,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:55,342 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 1bb2cde891fcf5bb7edaa5296fcaff7b#C#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:54:55,360 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/88c1adf21da14644b11b33bea2bf6ac0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/88c1adf21da14644b11b33bea2bf6ac0 2023-04-17 03:54:55,364 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/C of 1bb2cde891fcf5bb7edaa5296fcaff7b into 88c1adf21da14644b11b33bea2bf6ac0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:55,364 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:55,364 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/C, priority=11, startTime=1681703695295; duration=0sec 2023-04-17 03:54:55,364 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:55,691 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:55,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:55,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703755711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:55,723 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/5527b7d698964006953ad92f1b1ef439 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5527b7d698964006953ad92f1b1ef439 2023-04-17 03:54:55,727 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 1bb2cde891fcf5bb7edaa5296fcaff7b/A of 1bb2cde891fcf5bb7edaa5296fcaff7b into 5527b7d698964006953ad92f1b1ef439(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:54:55,727 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:55,727 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b., storeName=1bb2cde891fcf5bb7edaa5296fcaff7b/A, priority=11, startTime=1681703695295; duration=0sec 2023-04-17 03:54:55,727 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:54:55,750 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/d8ecd50cd48541ccbeab4cc4e63b8548 2023-04-17 03:54:55,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703755813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:56,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703756017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,160 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/8c0a3ca29fb643a0b373c90964fd4175 2023-04-17 03:54:56,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703756318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,574 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/baaf00c1730e468aa968dfaf1014d0a5 2023-04-17 03:54:56,578 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/d8ecd50cd48541ccbeab4cc4e63b8548 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d8ecd50cd48541ccbeab4cc4e63b8548 2023-04-17 03:54:56,582 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d8ecd50cd48541ccbeab4cc4e63b8548, entries=150, sequenceid=362, filesize=11.9 K 2023-04-17 03:54:56,582 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/8c0a3ca29fb643a0b373c90964fd4175 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8c0a3ca29fb643a0b373c90964fd4175 2023-04-17 03:54:56,585 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8c0a3ca29fb643a0b373c90964fd4175, entries=150, sequenceid=362, filesize=11.9 K 2023-04-17 03:54:56,586 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/baaf00c1730e468aa968dfaf1014d0a5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/baaf00c1730e468aa968dfaf1014d0a5 2023-04-17 03:54:56,589 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/baaf00c1730e468aa968dfaf1014d0a5, entries=150, sequenceid=362, filesize=11.9 K 2023-04-17 03:54:56,590 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1260ms, sequenceid=362, compaction requested=false 2023-04-17 03:54:56,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:56,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:56,592 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4257d329[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:56,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:56,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:56,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:56,597 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:56,599 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:56,599 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:56,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,603 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,604 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:56,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:56,610 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:56,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:56,611 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,611 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:56,611 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:56,711 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:56,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:56,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:56,714 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:56,714 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:56,714 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:56,714 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:56,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:56,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,716 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:56,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:56,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:56,717 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:56,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:56,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:56,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:56,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:56,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:56,718 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:56,718 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:56,718 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:56,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:56,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:56,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:56,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703696849, max timestamp=1681703696872Num uniques 859; 2023-04-17 03:54:56,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703696849, max timestamp=1681703696872Num uniques 859; 2023-04-17 03:54:56,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:56,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:56,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703696849, max timestamp=1681703696872Num uniques 859; 2023-04-17 03:54:56,874 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:56,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:56,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703756923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703757026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,128 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/a5ebb903f54d45f58c12ebf3e9b44d60 2023-04-17 03:54:57,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703757230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703757534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,539 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/bf7bb93bfcf24a4aa7282eef0bdce067 2023-04-17 03:54:57,549 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/50e7020bad1b470ca74cf8a97d4bf0b8 2023-04-17 03:54:57,553 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/a5ebb903f54d45f58c12ebf3e9b44d60 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a5ebb903f54d45f58c12ebf3e9b44d60 2023-04-17 03:54:57,557 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a5ebb903f54d45f58c12ebf3e9b44d60, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:54:57,558 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/bf7bb93bfcf24a4aa7282eef0bdce067 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bf7bb93bfcf24a4aa7282eef0bdce067 2023-04-17 03:54:57,562 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bf7bb93bfcf24a4aa7282eef0bdce067, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:54:57,563 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/50e7020bad1b470ca74cf8a97d4bf0b8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/50e7020bad1b470ca74cf8a97d4bf0b8 2023-04-17 03:54:57,567 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/50e7020bad1b470ca74cf8a97d4bf0b8, entries=150, sequenceid=375, filesize=11.9 K 2023-04-17 03:54:57,568 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 850ms, sequenceid=375, compaction requested=true 2023-04-17 03:54:57,568 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:57,568 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:57,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:57,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf05d85[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:57,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:57,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:57,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:57,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:57,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:57,583 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:57,583 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:57,583 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:57,583 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:57,583 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:57,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:57,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:57,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:57,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:57,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:57,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:57,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:57,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:57,589 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:57,589 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:57,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:57,689 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:57,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:57,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:57,692 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:57,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:57,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:57,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:57,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:57,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:57,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:57,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:57,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:57,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:57,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:57,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:57,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:57,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:57,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:57,697 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:57,697 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:54:58,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:58,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:58,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:58,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703758064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,105 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/a20697945cab456f878097475e8a1396 2023-04-17 03:54:58,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:58,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703758167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:58,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703758369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,516 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/73ef5d65c5ac41a397b730b53ce05f6c 2023-04-17 03:54:58,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:58,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703758671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,927 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/3be4ff2fe0c44a92885091fee3b3c6c9 2023-04-17 03:54:58,931 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/a20697945cab456f878097475e8a1396 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a20697945cab456f878097475e8a1396 2023-04-17 03:54:58,934 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a20697945cab456f878097475e8a1396, entries=150, sequenceid=411, filesize=11.9 K 2023-04-17 03:54:58,935 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/73ef5d65c5ac41a397b730b53ce05f6c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/73ef5d65c5ac41a397b730b53ce05f6c 2023-04-17 03:54:58,938 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/73ef5d65c5ac41a397b730b53ce05f6c, entries=150, sequenceid=411, filesize=11.9 K 2023-04-17 03:54:58,939 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/3be4ff2fe0c44a92885091fee3b3c6c9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3be4ff2fe0c44a92885091fee3b3c6c9 2023-04-17 03:54:58,942 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3be4ff2fe0c44a92885091fee3b3c6c9, entries=150, sequenceid=411, filesize=11.9 K 2023-04-17 03:54:58,942 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1245ms, sequenceid=411, compaction requested=true 2023-04-17 03:54:58,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:58,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:58,943 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,944 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:58,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:58,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45b046ba[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:58,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:58,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,947 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:58,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:58,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:58,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:58,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:58,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:58,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:58,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:58,951 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:58,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:58,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:58,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:58,951 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:58,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:58,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:58,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:58,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:58,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:58,956 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:58,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:58,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:58,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:58,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:58,958 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:58,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:58,959 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:58,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,959 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:58,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:58,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,059 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:54:59,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:54:59,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:54:59,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:54:59,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:54:59,063 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:54:59,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:59,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:54:59,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:54:59,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,064 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:59,065 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,065 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:59,065 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,065 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:59,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:59,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:54:59,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:54:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:54:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:54:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:54:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:54:59,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:54:59,066 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:59,066 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:54:59,066 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:54:59,067 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:54:59,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:54:59,176 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703699213, max timestamp=1681703699230Num uniques 859; 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703699213, max timestamp=1681703699230Num uniques 859; 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703699213, max timestamp=1681703699230Num uniques 859; 2023-04-17 03:54:59,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:54:59,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:59,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703759284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:59,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703759388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,473 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/8bbf89fac9f946e394f3d1d87392d51d 2023-04-17 03:54:59,483 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/27ab44560ac4441794148cb41547e91a 2023-04-17 03:54:59,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:59,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703759591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,892 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/3ec42f9e83a242aabea346b9d37fecdc 2023-04-17 03:54:59,896 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/8bbf89fac9f946e394f3d1d87392d51d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8bbf89fac9f946e394f3d1d87392d51d 2023-04-17 03:54:59,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:54:59,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703759896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,899 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8bbf89fac9f946e394f3d1d87392d51d, entries=150, sequenceid=422, filesize=11.9 K 2023-04-17 03:54:59,900 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/27ab44560ac4441794148cb41547e91a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/27ab44560ac4441794148cb41547e91a 2023-04-17 03:54:59,903 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/27ab44560ac4441794148cb41547e91a, entries=150, sequenceid=422, filesize=11.9 K 2023-04-17 03:54:59,904 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/3ec42f9e83a242aabea346b9d37fecdc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3ec42f9e83a242aabea346b9d37fecdc 2023-04-17 03:54:59,907 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3ec42f9e83a242aabea346b9d37fecdc, entries=150, sequenceid=422, filesize=11.9 K 2023-04-17 03:54:59,908 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 842ms, sequenceid=422, compaction requested=true 2023-04-17 03:54:59,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:54:59,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:54:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:54:59,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:54:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:54:59,909 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:54:59,910 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,910 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@714848f1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:54:59,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:54:59,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,915 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,915 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:54:59,915 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:54:59,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:54:59,915 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:54:59,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:54:59,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:59,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:54:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:59,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:59,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:54:59,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:54:59,919 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:54:59,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:54:59,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:54:59,919 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:54:59,919 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:54:59,921 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,921 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:54:59,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,921 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:59,921 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:54:59,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:54:59,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:54:59,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:54:59,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:54:59,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:54:59,925 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:54:59,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:54:59,925 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:54:59,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:54:59,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:54:59,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:54:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:54:59,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:54:59,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:00,025 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:00,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:00,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:00,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:00,029 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:00,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:00,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:00,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:00,030 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:00,030 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:00,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:00,032 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:00,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:00,032 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:00,032 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:00,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:00,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:00,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:00,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. started... 2023-04-17 03:55:00,034 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-17 03:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:55:00,035 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:00,035 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:55:00,035 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:00,035 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:55:00,035 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:00,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. as already flushing 2023-04-17 03:55:00,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:00,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703760428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:00,449 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/df419b22d7bb465f950010c166ebf796 2023-04-17 03:55:00,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703760532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:00,604 DEBUG [Thread-1674] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:60620 2023-04-17 03:55:00,604 DEBUG [Thread-1672] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:60620 2023-04-17 03:55:00,604 DEBUG [Thread-1674] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:00,604 DEBUG [Thread-1676] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:60620 2023-04-17 03:55:00,604 DEBUG [Thread-1672] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:00,604 DEBUG [Thread-1676] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:00,607 DEBUG [Thread-1678] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f8ff99f to 127.0.0.1:60620 2023-04-17 03:55:00,607 DEBUG [Thread-1678] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:00,608 DEBUG [Thread-1670] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:60620 2023-04-17 03:55:00,608 DEBUG [Thread-1670] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:00,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:00,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703760735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:00,859 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/410fcbec5d184b32847fec189c900927 2023-04-17 03:55:01,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:01,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35490 deadline: 1681703761039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,268 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e670a0eb2039455abd05eebda03cdbf0 2023-04-17 03:55:01,271 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/df419b22d7bb465f950010c166ebf796 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/df419b22d7bb465f950010c166ebf796 2023-04-17 03:55:01,274 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/df419b22d7bb465f950010c166ebf796, entries=150, sequenceid=458, filesize=11.9 K 2023-04-17 03:55:01,275 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/410fcbec5d184b32847fec189c900927 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/410fcbec5d184b32847fec189c900927 2023-04-17 03:55:01,278 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/410fcbec5d184b32847fec189c900927, entries=150, sequenceid=458, filesize=11.9 K 2023-04-17 03:55:01,278 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/e670a0eb2039455abd05eebda03cdbf0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e670a0eb2039455abd05eebda03cdbf0 2023-04-17 03:55:01,281 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e670a0eb2039455abd05eebda03cdbf0, entries=150, sequenceid=458, filesize=11.9 K 2023-04-17 03:55:01,282 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1248ms, sequenceid=458, compaction requested=true 2023-04-17 03:55:01,282 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:55:01,282 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:01,282 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:01,291 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:01,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfa136a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:01,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:01,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:01,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:01,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:01,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:01,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:01,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:01,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:01,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:01,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:01,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:01,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:01,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:01,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:01,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:01,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:01,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:01,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:01,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:01,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:01,303 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,303 DEBUG [Thread-1669] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:01,303 DEBUG [Thread-1669] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:01,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:01,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:01,403 DEBUG [Thread-1669] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:01,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:01,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:01,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:55:01,543 DEBUG [Thread-1667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:60620 2023-04-17 03:55:01,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:55:01,543 DEBUG [Thread-1667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:01,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:01,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:55:01,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:01,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:55:01,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:01,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2c895f6b527c42b680c0213266cba40b 2023-04-17 03:55:02,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/25750d39f21e4823a390dc5cbeea1a92 2023-04-17 03:55:02,556 DEBUG [Thread-1663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:60620 2023-04-17 03:55:02,556 DEBUG [Thread-1663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:02,566 DEBUG [Thread-1665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:60620 2023-04-17 03:55:02,566 DEBUG [Thread-1665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:02,573 DEBUG [Thread-1659] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:60620 2023-04-17 03:55:02,573 DEBUG [Thread-1659] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:02,617 DEBUG [Thread-1661] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:60620 2023-04-17 03:55:02,617 DEBUG [Thread-1661] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:02,617 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 18 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 301 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 6236 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 6143 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 5996 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 6222 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 6159 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:55:02,618 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:55:02,618 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:60620 2023-04-17 03:55:02,618 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:02,619 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:55:02,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:55:02,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:02,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 03:55:02,622 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703702622"}]},"ts":"1681703702622"} 2023-04-17 03:55:02,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:55:02,625 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:55:02,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, UNASSIGN}] 2023-04-17 03:55:02,627 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, UNASSIGN 2023-04-17 03:55:02,627 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=1bb2cde891fcf5bb7edaa5296fcaff7b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:02,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703702627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703702627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703702627"}]},"ts":"1681703702627"} 2023-04-17 03:55:02,628 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:02,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 03:55:02,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/231daf17762a47cbbe33983788311c52 2023-04-17 03:55:02,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/2c895f6b527c42b680c0213266cba40b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c895f6b527c42b680c0213266cba40b 2023-04-17 03:55:02,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c895f6b527c42b680c0213266cba40b, entries=150, sequenceid=470, filesize=11.9 K 2023-04-17 03:55:02,774 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/25750d39f21e4823a390dc5cbeea1a92 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/25750d39f21e4823a390dc5cbeea1a92 2023-04-17 03:55:02,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/25750d39f21e4823a390dc5cbeea1a92, entries=150, sequenceid=470, filesize=11.9 K 2023-04-17 03:55:02,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/231daf17762a47cbbe33983788311c52 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/231daf17762a47cbbe33983788311c52 2023-04-17 03:55:02,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:02,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1bb2cde891fcf5bb7edaa5296fcaff7b, disabling compactions & flushes 2023-04-17 03:55:02,780 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:02,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/231daf17762a47cbbe33983788311c52, entries=150, sequenceid=470, filesize=11.9 K 2023-04-17 03:55:02,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 1239ms, sequenceid=470, compaction requested=true 2023-04-17 03:55:02,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:55:02,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:02,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:02,782 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. because compaction request was cancelled 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:02,782 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. because compaction request was cancelled 2023-04-17 03:55:02,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. after waiting 0 ms 2023-04-17 03:55:02,782 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. because compaction request was cancelled 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:02,782 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:02,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1bb2cde891fcf5bb7edaa5296fcaff7b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=A 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=B 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 1bb2cde891fcf5bb7edaa5296fcaff7b, store=C 2023-04-17 03:55:02,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:02,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/9f3b8db41d0e4bad93ca3204b8866b43 2023-04-17 03:55:02,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 03:55:03,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/007667d0184a42fc9002eebe301e5560 2023-04-17 03:55:03,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 03:55:03,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/28001922886f432b840054be53c05e78 2023-04-17 03:55:03,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/A/9f3b8db41d0e4bad93ca3204b8866b43 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9f3b8db41d0e4bad93ca3204b8866b43 2023-04-17 03:55:03,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9f3b8db41d0e4bad93ca3204b8866b43, entries=150, sequenceid=477, filesize=11.9 K 2023-04-17 03:55:03,610 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/B/007667d0184a42fc9002eebe301e5560 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/007667d0184a42fc9002eebe301e5560 2023-04-17 03:55:03,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/007667d0184a42fc9002eebe301e5560, entries=150, sequenceid=477, filesize=11.9 K 2023-04-17 03:55:03,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/.tmp/C/28001922886f432b840054be53c05e78 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/28001922886f432b840054be53c05e78 2023-04-17 03:55:03,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/28001922886f432b840054be53c05e78, entries=150, sequenceid=477, filesize=11.9 K 2023-04-17 03:55:03,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 1bb2cde891fcf5bb7edaa5296fcaff7b in 835ms, sequenceid=477, compaction requested=true 2023-04-17 03:55:03,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/6940a457ac6b459491ef6da87d5d13ec, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9420ebf28dc244889f1445dafd94ef82, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5faad08dba5142f3991ea0b5502061db, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/653da77fa6b4413b831def56c717e5cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b3d8c1dcc0884b119ec6009b305df723, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04] to archive 2023-04-17 03:55:03,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:03,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/f9d12bdae4814467b6d6d1ec97d27766 2023-04-17 03:55:03,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2f22ba2d58ad4d5c97ae00adec9a8f7d 2023-04-17 03:55:03,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/6940a457ac6b459491ef6da87d5d13ec to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/6940a457ac6b459491ef6da87d5d13ec 2023-04-17 03:55:03,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/024c8973af924270a387e2e2c85067c6 2023-04-17 03:55:03,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8767b35073c0451f8b5e6267d38ce73b 2023-04-17 03:55:03,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e1c3f5cc7f024fbfb885aeb3ca562659 2023-04-17 03:55:03,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9420ebf28dc244889f1445dafd94ef82 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9420ebf28dc244889f1445dafd94ef82 2023-04-17 03:55:03,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d2e11bc1f8d3495baef497d009346947 2023-04-17 03:55:03,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c3e3635de444eb68d92c7d59a3917b1 2023-04-17 03:55:03,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/079a6f64d1694406b4fd12e93855dc77 2023-04-17 03:55:03,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5faad08dba5142f3991ea0b5502061db to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5faad08dba5142f3991ea0b5502061db 2023-04-17 03:55:03,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/113d4a1c2529433992cec986baafa299 2023-04-17 03:55:03,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/653da77fa6b4413b831def56c717e5cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/653da77fa6b4413b831def56c717e5cb 2023-04-17 03:55:03,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/fff44f43f2d94358a8d75c2931b86e26 2023-04-17 03:55:03,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e42b4116d954442496d927a86a593c5d 2023-04-17 03:55:03,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b78d0b4d442143309402dc4ef9414f7f 2023-04-17 03:55:03,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b3d8c1dcc0884b119ec6009b305df723 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/b3d8c1dcc0884b119ec6009b305df723 2023-04-17 03:55:03,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/c9cece6777d4447c995fb820258aeafe 2023-04-17 03:55:03,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/dced4301c3234b0bbfc9f46d1e1edb00 2023-04-17 03:55:03,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/0f5080b4a6594662873e92bd9f4c2e6c 2023-04-17 03:55:03,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/e005fff346d741009d8d845fd63e2ade 2023-04-17 03:55:03,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/192082a2477f4d54b16529895ade0a04 2023-04-17 03:55:03,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/39493a5516894d02bafcc5d3f4b6c317, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/3ba05fab013047aebd3e9f8319f6e8c8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d839c67fdc75487cadc3a76107ab4251, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9c106cbed95e4efea2c9fcd40d7e3686, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/5276e4c1990b487797519f78838da3ea, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7] to archive 2023-04-17 03:55:03,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:03,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/889b120292094e0ca1f1044949e8caed 2023-04-17 03:55:03,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d190de4bf3cb487e92b7f6baebe9e73c 2023-04-17 03:55:03,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/39493a5516894d02bafcc5d3f4b6c317 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/39493a5516894d02bafcc5d3f4b6c317 2023-04-17 03:55:03,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/88dc3341cad241d484419c6b2a779446 2023-04-17 03:55:03,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/e44d0822f849419ab87ebff30114e58a 2023-04-17 03:55:03,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/3ba05fab013047aebd3e9f8319f6e8c8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/3ba05fab013047aebd3e9f8319f6e8c8 2023-04-17 03:55:03,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8682783f484944be9df9552424f31d7e 2023-04-17 03:55:03,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/503ed18e1c004e36b747eaaf1f829eae 2023-04-17 03:55:03,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/15f3156907cd41e4a4747e471b4d7c92 2023-04-17 03:55:03,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d839c67fdc75487cadc3a76107ab4251 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d839c67fdc75487cadc3a76107ab4251 2023-04-17 03:55:03,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/d5fc65b07725419ba3debc30266e7153 2023-04-17 03:55:03,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/ad6d284580ad4a5d8772ff7e350c2d0f 2023-04-17 03:55:03,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9c106cbed95e4efea2c9fcd40d7e3686 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9c106cbed95e4efea2c9fcd40d7e3686 2023-04-17 03:55:03,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/0bc4730ffdb840b58d2579905ccc4cac 2023-04-17 03:55:03,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/384879b9714442d78d7d1a479a048ed4 2023-04-17 03:55:03,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bbbec1ed5c41427b900afb70747f8bf9 2023-04-17 03:55:03,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/5276e4c1990b487797519f78838da3ea to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/5276e4c1990b487797519f78838da3ea 2023-04-17 03:55:03,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/42ed926b18f8433ba237ab6bb9f601cb 2023-04-17 03:55:03,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/9d43b1e87c53481e81e24ba87cd84dc7 2023-04-17 03:55:03,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/90c1e7a6a2f7401b8b3982fbb3d36c99 2023-04-17 03:55:03,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/cec90908c33c43b1b1ae803662f4a038 2023-04-17 03:55:03,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/c54a47b846f548e98c7b137f53406fa7 2023-04-17 03:55:03,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/77e4de9e3464450dbd00e324e4178f07, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/cd7898fc37674ef082ca5d6f259aee5f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/18517c894ace4df5995542e4b47e33f2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/bd1d78ab6f7c45eba7ffc3f0516471bd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8f474143dadd4feeab7ef14e85d55a54, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a] to archive 2023-04-17 03:55:03,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:03,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/63bb94d0057e480282b12b80c7841fc7 2023-04-17 03:55:03,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/1c93acdfada34a11b1e58ddccd5208e7 2023-04-17 03:55:03,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/77e4de9e3464450dbd00e324e4178f07 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/77e4de9e3464450dbd00e324e4178f07 2023-04-17 03:55:03,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d17622bf31554a62a5cc5a471cf6c008 2023-04-17 03:55:03,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d11116fc8b04ea5ae0cdff4d8be4140 2023-04-17 03:55:03,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/cd7898fc37674ef082ca5d6f259aee5f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/cd7898fc37674ef082ca5d6f259aee5f 2023-04-17 03:55:03,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/c39dc7115fa44dc18421f94d9598790d 2023-04-17 03:55:03,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/d278db8d6c0048cb9af0474fc10ab016 2023-04-17 03:55:03,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/2479e7899aae46c1b9b3fef446c28d85 2023-04-17 03:55:03,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/18517c894ace4df5995542e4b47e33f2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/18517c894ace4df5995542e4b47e33f2 2023-04-17 03:55:03,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/6b30a5bd7e5048fa82a6fee58764c7cb 2023-04-17 03:55:03,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8afcc94faaa84f178318570a4f80e720 2023-04-17 03:55:03,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/bd1d78ab6f7c45eba7ffc3f0516471bd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/bd1d78ab6f7c45eba7ffc3f0516471bd 2023-04-17 03:55:03,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/9503e206d8f94e6da7da52273084609a 2023-04-17 03:55:03,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/ad7af4fafba44939a8b07d57d2bf3758 2023-04-17 03:55:03,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/077bbaa874d8402780a6b4641e8c79fe 2023-04-17 03:55:03,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8f474143dadd4feeab7ef14e85d55a54 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8f474143dadd4feeab7ef14e85d55a54 2023-04-17 03:55:03,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/8d203f4b7f844770882d0326e80e1a4d 2023-04-17 03:55:03,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/f1276b40da68461c9a3f78f028c4532f 2023-04-17 03:55:03,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/a81f1569e31f4c91b037f2140e3b397b 2023-04-17 03:55:03,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e4dc3b045514428894674e82c818bef2 2023-04-17 03:55:03,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e1df866d12f04406b5351fdefb0ef21a 2023-04-17 03:55:03,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/recovered.edits/480.seqid, newMaxSeqId=480, maxSeqId=1 2023-04-17 03:55:03,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b. 2023-04-17 03:55:03,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1bb2cde891fcf5bb7edaa5296fcaff7b: 2023-04-17 03:55:03,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:03,708 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=1bb2cde891fcf5bb7edaa5296fcaff7b, regionState=CLOSED 2023-04-17 03:55:03,708 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703703708"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703703708"}]},"ts":"1681703703708"} 2023-04-17 03:55:03,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-17 03:55:03,710 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 1bb2cde891fcf5bb7edaa5296fcaff7b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 1.0810 sec 2023-04-17 03:55:03,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-17 03:55:03,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=1bb2cde891fcf5bb7edaa5296fcaff7b, UNASSIGN in 1.0840 sec 2023-04-17 03:55:03,711 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703703711"}]},"ts":"1681703703711"} 2023-04-17 03:55:03,712 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:55:03,713 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:55:03,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.0950 sec 2023-04-17 03:55:03,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-17 03:55:03,725 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-17 03:55:03,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:55:03,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,727 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 03:55:03,727 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,730 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:03,732 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/recovered.edits] 2023-04-17 03:55:03,734 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c895f6b527c42b680c0213266cba40b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/2c895f6b527c42b680c0213266cba40b 2023-04-17 03:55:03,735 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5527b7d698964006953ad92f1b1ef439 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/5527b7d698964006953ad92f1b1ef439 2023-04-17 03:55:03,736 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8bbf89fac9f946e394f3d1d87392d51d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/8bbf89fac9f946e394f3d1d87392d51d 2023-04-17 03:55:03,737 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9f3b8db41d0e4bad93ca3204b8866b43 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/9f3b8db41d0e4bad93ca3204b8866b43 2023-04-17 03:55:03,738 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a20697945cab456f878097475e8a1396 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a20697945cab456f878097475e8a1396 2023-04-17 03:55:03,739 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a5ebb903f54d45f58c12ebf3e9b44d60 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/a5ebb903f54d45f58c12ebf3e9b44d60 2023-04-17 03:55:03,740 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d8ecd50cd48541ccbeab4cc4e63b8548 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/d8ecd50cd48541ccbeab4cc4e63b8548 2023-04-17 03:55:03,741 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/df419b22d7bb465f950010c166ebf796 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/A/df419b22d7bb465f950010c166ebf796 2023-04-17 03:55:03,743 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/007667d0184a42fc9002eebe301e5560 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/007667d0184a42fc9002eebe301e5560 2023-04-17 03:55:03,745 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/06defb93105a46ecaa1fcebe8d046200 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/06defb93105a46ecaa1fcebe8d046200 2023-04-17 03:55:03,746 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/25750d39f21e4823a390dc5cbeea1a92 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/25750d39f21e4823a390dc5cbeea1a92 2023-04-17 03:55:03,747 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/27ab44560ac4441794148cb41547e91a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/27ab44560ac4441794148cb41547e91a 2023-04-17 03:55:03,748 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/410fcbec5d184b32847fec189c900927 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/410fcbec5d184b32847fec189c900927 2023-04-17 03:55:03,750 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/73ef5d65c5ac41a397b730b53ce05f6c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/73ef5d65c5ac41a397b730b53ce05f6c 2023-04-17 03:55:03,751 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8c0a3ca29fb643a0b373c90964fd4175 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/8c0a3ca29fb643a0b373c90964fd4175 2023-04-17 03:55:03,752 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bf7bb93bfcf24a4aa7282eef0bdce067 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/B/bf7bb93bfcf24a4aa7282eef0bdce067 2023-04-17 03:55:03,755 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/231daf17762a47cbbe33983788311c52 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/231daf17762a47cbbe33983788311c52 2023-04-17 03:55:03,756 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/28001922886f432b840054be53c05e78 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/28001922886f432b840054be53c05e78 2023-04-17 03:55:03,757 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3be4ff2fe0c44a92885091fee3b3c6c9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3be4ff2fe0c44a92885091fee3b3c6c9 2023-04-17 03:55:03,758 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3ec42f9e83a242aabea346b9d37fecdc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/3ec42f9e83a242aabea346b9d37fecdc 2023-04-17 03:55:03,759 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/50e7020bad1b470ca74cf8a97d4bf0b8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/50e7020bad1b470ca74cf8a97d4bf0b8 2023-04-17 03:55:03,759 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/88c1adf21da14644b11b33bea2bf6ac0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/88c1adf21da14644b11b33bea2bf6ac0 2023-04-17 03:55:03,760 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/baaf00c1730e468aa968dfaf1014d0a5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/baaf00c1730e468aa968dfaf1014d0a5 2023-04-17 03:55:03,761 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e670a0eb2039455abd05eebda03cdbf0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/C/e670a0eb2039455abd05eebda03cdbf0 2023-04-17 03:55:03,764 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/recovered.edits/480.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b/recovered.edits/480.seqid 2023-04-17 03:55:03,764 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/1bb2cde891fcf5bb7edaa5296fcaff7b 2023-04-17 03:55:03,764 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:03,766 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,767 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:55:03,769 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:55:03,769 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,770 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:55:03,770 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703703770"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:03,771 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:55:03,771 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1bb2cde891fcf5bb7edaa5296fcaff7b, NAME => 'TestAcidGuarantees,,1681703678438.1bb2cde891fcf5bb7edaa5296fcaff7b.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:55:03,771 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:55:03,771 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703703771"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:03,773 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:55:03,775 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,776 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 50 msec 2023-04-17 03:55:03,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-17 03:55:03,828 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-17 03:55:03,839 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 224) - Thread LEAK? -, OpenFileDescriptor=540 (was 525) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 312) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=656 (was 685) 2023-04-17 03:55:03,850 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=540, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=169, AvailableMemoryMB=656 2023-04-17 03:55:03,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:55:03,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:55:03,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:03,854 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:55:03,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-17 03:55:03,854 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:55:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:03,855 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:03,856 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 empty. 2023-04-17 03:55:03,856 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:03,856 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:03,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:04,263 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:55:04,264 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ed08334fa02b7639086fbab0bf0f1d10, NAME => 'TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:55:04,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ed08334fa02b7639086fbab0bf0f1d10, disabling compactions & flushes 2023-04-17 03:55:04,670 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. after waiting 0 ms 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,670 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,670 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:04,672 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:55:04,673 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703704673"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703704673"}]},"ts":"1681703704673"} 2023-04-17 03:55:04,674 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:55:04,675 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:55:04,675 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703704675"}]},"ts":"1681703704675"} 2023-04-17 03:55:04,675 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:55:04,678 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, ASSIGN}] 2023-04-17 03:55:04,680 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, ASSIGN 2023-04-17 03:55:04,680 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:55:04,831 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:04,831 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703704831"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703704831"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703704831"}]},"ts":"1681703704831"} 2023-04-17 03:55:04,833 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:04,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:04,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ed08334fa02b7639086fbab0bf0f1d10, NAME => 'TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,988 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,989 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A 2023-04-17 03:55:04,989 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A 2023-04-17 03:55:04,990 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:04,990 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName A 2023-04-17 03:55:04,990 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:04,990 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,991 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B 2023-04-17 03:55:04,991 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B 2023-04-17 03:55:04,991 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:04,992 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName B 2023-04-17 03:55:04,992 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:04,992 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,993 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C 2023-04-17 03:55:04,993 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C 2023-04-17 03:55:04,993 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:04,993 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName C 2023-04-17 03:55:04,994 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:04,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:04,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,996 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:55:04,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:04,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:55:04,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ed08334fa02b7639086fbab0bf0f1d10; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59175590, jitterRate=-0.11821499466896057}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:55:04,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:04,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., pid=37, masterSystemTime=1681703704984 2023-04-17 03:55:05,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:05,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:05,000 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:05,000 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703705000"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703705000"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703705000"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703705000"}]},"ts":"1681703705000"} 2023-04-17 03:55:05,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-17 03:55:05,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 in 169 msec 2023-04-17 03:55:05,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-17 03:55:05,003 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, ASSIGN in 324 msec 2023-04-17 03:55:05,004 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:55:05,004 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703705004"}]},"ts":"1681703705004"} 2023-04-17 03:55:05,005 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:55:05,007 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:55:05,008 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1550 sec 2023-04-17 03:55:05,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-17 03:55:05,958 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-17 03:55:05,959 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:05,966 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:05,967 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:05,969 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54494, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:05,970 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:55:05,971 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60454, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:55:05,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:55:05,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:55:05,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:06,382 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:55:06,383 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:55:06,383 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:55:06,385 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 03:55:06,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, REOPEN/MOVE}] 2023-04-17 03:55:06,388 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, REOPEN/MOVE 2023-04-17 03:55:06,389 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:06,389 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703706389"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703706389"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703706389"}]},"ts":"1681703706389"} 2023-04-17 03:55:06,390 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:06,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ed08334fa02b7639086fbab0bf0f1d10, disabling compactions & flushes 2023-04-17 03:55:06,542 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. after waiting 0 ms 2023-04-17 03:55:06,542 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,545 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 03:55:06,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:06,546 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ed08334fa02b7639086fbab0bf0f1d10 to self. 2023-04-17 03:55:06,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,547 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=CLOSED 2023-04-17 03:55:06,547 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703706547"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703706547"}]},"ts":"1681703706547"} 2023-04-17 03:55:06,549 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-17 03:55:06,549 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 in 158 msec 2023-04-17 03:55:06,550 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=true 2023-04-17 03:55:06,700 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:06,700 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703706700"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703706700"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703706700"}]},"ts":"1681703706700"} 2023-04-17 03:55:06,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:06,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ed08334fa02b7639086fbab0bf0f1d10, NAME => 'TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:55:06,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:06,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,856 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,857 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,857 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A 2023-04-17 03:55:06,857 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A 2023-04-17 03:55:06,858 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:06,858 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName A 2023-04-17 03:55:06,859 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:06,859 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,860 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B 2023-04-17 03:55:06,860 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B 2023-04-17 03:55:06,860 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:06,860 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName B 2023-04-17 03:55:06,861 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:06,861 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,862 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C 2023-04-17 03:55:06,862 DEBUG [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C 2023-04-17 03:55:06,862 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:06,862 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ed08334fa02b7639086fbab0bf0f1d10 columnFamilyName C 2023-04-17 03:55:06,863 INFO [StoreOpener-ed08334fa02b7639086fbab0bf0f1d10-1] regionserver.HStore(310): Store=ed08334fa02b7639086fbab0bf0f1d10/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:06,863 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,863 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,864 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:55:06,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ed08334fa02b7639086fbab0bf0f1d10; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61729089, jitterRate=-0.08016489446163177}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:55:06,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:06,868 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., pid=42, masterSystemTime=1681703706852 2023-04-17 03:55:06,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,869 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:06,869 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703706869"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703706869"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703706869"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703706869"}]},"ts":"1681703706869"} 2023-04-17 03:55:06,871 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-17 03:55:06,871 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 in 169 msec 2023-04-17 03:55:06,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-17 03:55:06,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, REOPEN/MOVE in 484 msec 2023-04-17 03:55:06,873 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-17 03:55:06,874 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-17 03:55:06,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 902 msec 2023-04-17 03:55:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-17 03:55:06,876 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,880 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,881 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,884 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,884 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,889 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,889 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,892 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,893 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,896 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,896 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,899 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,900 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,904 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,905 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,910 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,910 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,913 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,913 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5ce79633 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:06,917 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@159c5e0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:06,922 DEBUG [hconnection-0x64821d66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,922 DEBUG [hconnection-0x3cb9f896-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:06,922 DEBUG [hconnection-0x4adb61bf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,923 DEBUG [hconnection-0x79927ed3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,923 DEBUG [hconnection-0x3b13b2a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,923 DEBUG [hconnection-0x1c84fd25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,923 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,924 DEBUG [hconnection-0x414e4293-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,924 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54518, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,925 DEBUG [hconnection-0x1ee7999-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,926 DEBUG [hconnection-0x5cd5b84e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,926 DEBUG [hconnection-0x325bb8ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:06,926 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54504, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,927 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54554, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,928 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,926 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54510, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,929 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,931 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,931 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54542, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,934 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54564, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:06,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:06,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:06,938 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:06,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:06,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:06,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:06,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:06,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:06,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:06,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:06,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:06,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:06,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:06,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:06,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:06,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:06,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:06,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:06,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:06,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:06,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:06,950 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:06,968 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:06,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:06,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:06,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703706979, max timestamp=1681703706984Num uniques 859; 2023-04-17 03:55:06,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703706979, max timestamp=1681703706984Num uniques 859; 2023-04-17 03:55:06,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703706979, max timestamp=1681703706984Num uniques 859; 2023-04-17 03:55:06,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:06,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:06,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:06,988 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304175cf272ace2f0457a947ce7cd9f7d6bfd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175cf272ace2f0457a947ce7cd9f7d6bfd 2023-04-17 03:55:07,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681703706987, max timestamp=1681703707007Num uniques 1018; 2023-04-17 03:55:07,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-17 03:55:07,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681703706987, max timestamp=1681703707007Num uniques 1018; 2023-04-17 03:55:07,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-17 03:55:07,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681703706987, max timestamp=1681703707007Num uniques 1018; 2023-04-17 03:55:07,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-17 03:55:07,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703767030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703767030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703767131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703767131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703767336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703767336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,399 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/e3ca79debf8c4056b4a7e4ebc83f4bee 2023-04-17 03:55:07,409 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/d926f6dc2eda4f7a837fce81c83703d5 2023-04-17 03:55:07,420 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/bff919aaa830489a872bb89346fb4ddc 2023-04-17 03:55:07,423 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/e3ca79debf8c4056b4a7e4ebc83f4bee as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee 2023-04-17 03:55:07,430 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee, entries=100, sequenceid=13, filesize=18.4 K 2023-04-17 03:55:07,430 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/d926f6dc2eda4f7a837fce81c83703d5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5 2023-04-17 03:55:07,434 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 03:55:07,435 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/bff919aaa830489a872bb89346fb4ddc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc 2023-04-17 03:55:07,438 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc, entries=100, sequenceid=13, filesize=9.3 K 2023-04-17 03:55:07,438 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for ed08334fa02b7639086fbab0bf0f1d10 in 488ms, sequenceid=13, compaction requested=false 2023-04-17 03:55:07,438 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:07,438 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:07,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:07,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:07,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:07,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:07,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:07,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:07,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:07,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14ff7b25[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:07,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:07,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,443 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,443 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:07,443 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:07,443 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:07,443 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,443 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:07,445 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:07,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:07,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:07,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:07,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:07,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:07,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:07,447 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:07,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:07,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:07,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:07,447 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:07,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:07,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,450 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:07,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:07,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:07,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:07,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,453 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:07,456 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:07,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:07,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:07,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:07,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:07,456 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:07,456 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:07,456 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:07,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:07,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:07,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:07,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:07,556 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:07,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:07,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:07,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:07,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:07,560 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:07,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:07,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:07,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:07,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:07,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,562 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:07,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:07,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:07,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:07,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:07,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:07,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:07,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:07,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:07,563 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:07,563 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:07,563 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=154.31 KB heapSize=429.84 KB 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:07,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:07,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:07,619 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:07,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703767640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703767640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703767930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703767937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:07,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703767937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:07,986 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d829f4fdc49b439ab222d923a5981ea9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d829f4fdc49b439ab222d923a5981ea9 2023-04-17 03:55:08,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703768145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703768148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703768235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703768241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,243 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703768241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,391 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=67, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/0b7589ba21974eb5bc9c99a4615a6369 2023-04-17 03:55:08,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703768738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703768745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:08,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703768745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,803 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9400336a6e7c49a1ae51d1dc893d5289 2023-04-17 03:55:08,816 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/1e2c4d65904741a09ee93f2b37a1cce9 2023-04-17 03:55:08,820 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/0b7589ba21974eb5bc9c99a4615a6369 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369 2023-04-17 03:55:08,824 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369, entries=150, sequenceid=67, filesize=25.1 K 2023-04-17 03:55:08,825 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9400336a6e7c49a1ae51d1dc893d5289 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289 2023-04-17 03:55:08,830 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289, entries=150, sequenceid=67, filesize=11.6 K 2023-04-17 03:55:08,831 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/1e2c4d65904741a09ee93f2b37a1cce9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9 2023-04-17 03:55:08,837 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9, entries=150, sequenceid=67, filesize=11.6 K 2023-04-17 03:55:08,838 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~429.80 KB/440112, currentSize=33.54 KB/34350 for ed08334fa02b7639086fbab0bf0f1d10 in 1275ms, sequenceid=67, compaction requested=false 2023-04-17 03:55:08,838 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:08,838 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:08,838 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:08,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:08,844 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,844 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@533d6a96[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:08,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:08,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:08,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:08,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:08,848 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:08,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:08,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:08,850 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:08,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:08,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:08,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:08,850 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:08,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:08,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:08,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,854 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:08,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:08,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:08,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:08,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:08,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:08,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:08,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:08,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:08,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:08,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:08,859 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:08,859 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:08,959 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:08,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:08,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:08,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:08,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:08,962 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:08,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:08,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:08,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:08,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:08,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:08,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:08,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:08,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:08,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:08,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:08,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:08,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:08,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:08,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:08,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:08,966 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:08,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:09,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:09,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:09,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703709183, max timestamp=1681703709205Num uniques 859; 2023-04-17 03:55:09,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703709183, max timestamp=1681703709205Num uniques 859; 2023-04-17 03:55:09,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703709183, max timestamp=1681703709205Num uniques 859; 2023-04-17 03:55:09,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:09,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:09,207 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:09,234 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703709248, max timestamp=1681703709270Num uniques 1018; 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703709248, max timestamp=1681703709270Num uniques 1018; 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703709248, max timestamp=1681703709270Num uniques 1018; 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:09,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703709308, max timestamp=1681703709327Num uniques 1077; 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703709308, max timestamp=1681703709327Num uniques 1077; 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703709308, max timestamp=1681703709327Num uniques 1077; 2023-04-17 03:55:09,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:09,380 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171f5b8fa06cb5435c82b72645aa85412f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f5b8fa06cb5435c82b72645aa85412f 2023-04-17 03:55:09,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703709358, max timestamp=1681703709393Num uniques 1086; 2023-04-17 03:55:09,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703709358, max timestamp=1681703709393Num uniques 1086; 2023-04-17 03:55:09,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-17 03:55:09,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-17 03:55:09,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703709358, max timestamp=1681703709393Num uniques 1086; 2023-04-17 03:55:09,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-17 03:55:09,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703709410, max timestamp=1681703709449Num uniques 1095; 2023-04-17 03:55:09,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 03:55:09,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703709410, max timestamp=1681703709449Num uniques 1095; 2023-04-17 03:55:09,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 03:55:09,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703709410, max timestamp=1681703709449Num uniques 1095; 2023-04-17 03:55:09,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-17 03:55:09,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703769496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703709484, max timestamp=1681703709499Num uniques 1054; 2023-04-17 03:55:09,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-17 03:55:09,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703709484, max timestamp=1681703709499Num uniques 1054; 2023-04-17 03:55:09,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-17 03:55:09,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703709484, max timestamp=1681703709499Num uniques 1054; 2023-04-17 03:55:09,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681703709550, max timestamp=1681703709583Num uniques 1013; 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681703709550, max timestamp=1681703709583Num uniques 1013; 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681703709550, max timestamp=1681703709583Num uniques 1013; 2023-04-17 03:55:09,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703709625, max timestamp=1681703709635Num uniques 1072; 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703709625, max timestamp=1681703709635Num uniques 1072; 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703709625, max timestamp=1681703709635Num uniques 1072; 2023-04-17 03:55:09,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=12452, suffix heap size=139456,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=272 2023-04-17 03:55:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703769688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703769689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703769751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703769754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703769758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,785 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/5ce68b498d144ad4bec8334eeaae9c3a 2023-04-17 03:55:09,795 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/adbfb1ad76044a929b0487af7e787ad2 2023-04-17 03:55:09,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703769793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:09,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703769794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,803 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a3aa5f6827054fcb9f17d0fd2fec3bba 2023-04-17 03:55:09,806 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/5ce68b498d144ad4bec8334eeaae9c3a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a 2023-04-17 03:55:09,810 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a, entries=150, sequenceid=75, filesize=25.1 K 2023-04-17 03:55:09,810 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/adbfb1ad76044a929b0487af7e787ad2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2 2023-04-17 03:55:09,813 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2, entries=150, sequenceid=75, filesize=11.6 K 2023-04-17 03:55:09,814 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a3aa5f6827054fcb9f17d0fd2fec3bba as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba 2023-04-17 03:55:09,817 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba, entries=150, sequenceid=75, filesize=11.6 K 2023-04-17 03:55:09,818 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for ed08334fa02b7639086fbab0bf0f1d10 in 852ms, sequenceid=75, compaction requested=true 2023-04-17 03:55:09,818 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:09,818 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:09,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:09,819 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,819 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:09,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:09,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:09,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:09,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6001afc0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:09,821 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:09,821 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,828 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:09,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:09,828 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:09,830 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:09,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:09,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:09,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:09,833 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:09,833 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:09,833 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:09,833 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:09,833 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:09,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:09,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:09,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:09,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:09,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:09,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:09,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:09,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:09,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,840 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:09,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:09,840 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:09,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:09,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:09,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:09,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:09,940 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:09,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:09,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:09,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:09,943 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:09,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:09,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:09,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:09,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:09,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:09,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:09,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:09,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:09,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:09,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:09,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:09,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:09,948 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=147.60 KB heapSize=414.23 KB 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:09,948 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:09,959 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417eef7044a415f4cebaada8b8105d85956 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eef7044a415f4cebaada8b8105d85956 2023-04-17 03:55:10,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:10,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:10,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703770011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703770015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703770122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703770123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703770326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703770329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,364 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/79051e3f49224de9b752771cf6225207 2023-04-17 03:55:10,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703770631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:10,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54556 deadline: 1681703770634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:10,782 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9dc298fcec754156b27f8a30ff27a2ae 2023-04-17 03:55:11,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:11,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703771139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:11,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54556 deadline: 1681703771141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,192 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/cfaf1126ad294bf7a938d166df34e33b 2023-04-17 03:55:11,198 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/79051e3f49224de9b752771cf6225207 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207 2023-04-17 03:55:11,202 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207, entries=150, sequenceid=232, filesize=25.3 K 2023-04-17 03:55:11,203 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9dc298fcec754156b27f8a30ff27a2ae as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae 2023-04-17 03:55:11,206 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae, entries=150, sequenceid=232, filesize=11.8 K 2023-04-17 03:55:11,207 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/cfaf1126ad294bf7a938d166df34e33b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b 2023-04-17 03:55:11,219 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b, entries=150, sequenceid=232, filesize=11.8 K 2023-04-17 03:55:11,219 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.19 KB/424128, currentSize=33.54 KB/34350 for ed08334fa02b7639086fbab0bf0f1d10 in 1271ms, sequenceid=232, compaction requested=true 2023-04-17 03:55:11,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:11,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:11,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:11,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:11,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:11,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:11,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbd81e9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:11,224 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:11,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:11,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:11,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:11,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:11,227 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:11,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:11,230 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:11,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:11,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:11,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:11,230 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:11,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,234 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:11,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:11,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,237 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:11,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,237 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:11,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:11,337 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:11,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:11,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:11,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:11,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:11,341 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:11,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:11,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:11,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:11,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,342 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:11,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:11,343 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,343 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:11,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:11,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:11,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:11,345 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:11,345 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:11,370 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417cd3cf5dd476b4f34a2637e5f0367893e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cd3cf5dd476b4f34a2637e5f0367893e 2023-04-17 03:55:11,379 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/16634477cec944bc9fb2161466adb5ef 2023-04-17 03:55:11,391 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/463d00d649664ed0ad2a2d52a9d46533 2023-04-17 03:55:11,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:11,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:11,802 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/c013206c74c64d7b8cfe5dc18e2c855f 2023-04-17 03:55:11,806 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/16634477cec944bc9fb2161466adb5ef as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef 2023-04-17 03:55:11,813 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef, entries=150, sequenceid=240, filesize=25.3 K 2023-04-17 03:55:11,814 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/463d00d649664ed0ad2a2d52a9d46533 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533 2023-04-17 03:55:11,818 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533, entries=150, sequenceid=240, filesize=11.8 K 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703711796, max timestamp=1681703711817Num uniques 859; 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703711796, max timestamp=1681703711817Num uniques 859; 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703711796, max timestamp=1681703711817Num uniques 859; 2023-04-17 03:55:11,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:11,819 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/c013206c74c64d7b8cfe5dc18e2c855f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f 2023-04-17 03:55:11,822 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f, entries=150, sequenceid=240, filesize=11.8 K 2023-04-17 03:55:11,823 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=40.25 KB/41220 for ed08334fa02b7639086fbab0bf0f1d10 in 479ms, sequenceid=240, compaction requested=true 2023-04-17 03:55:11,823 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:11,823 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:11,825 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5274e328[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:11,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:11,827 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,829 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:11,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:11,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-17 03:55:11,833 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,833 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:11,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:11,840 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:11,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:11,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:11,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:11,840 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:11,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,844 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:11,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:11,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,846 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:11,846 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:11,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703711868, max timestamp=1681703711887Num uniques 859; 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703711868, max timestamp=1681703711887Num uniques 859; 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703711868, max timestamp=1681703711887Num uniques 859; 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:11,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:11,946 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:11,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:11,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:11,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:11,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:11,951 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:11,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:11,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:11,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:11,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,953 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:11,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:11,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:11,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:11,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:11,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:11,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:11,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:11,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:11,956 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:11,957 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,958 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa55a2a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:11,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:11,959 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:11,960 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:11,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:11,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:11,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:11,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703771953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:11,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:11,971 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:11,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:11,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:11,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:11,971 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:11,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:11,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:11,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:11,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:11,978 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:11,978 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:11,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:11,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:11,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703771965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:11,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:11,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703771966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703772066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,078 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,081 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,083 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,084 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,084 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,085 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,085 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,085 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,085 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703772080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703772083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b79711[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,088 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,089 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,090 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,094 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,095 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,099 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,102 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,102 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703772150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54556 deadline: 1681703772150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,202 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,205 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,207 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,210 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,210 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,210 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,210 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,214 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,214 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5654fcc7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,215 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,215 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,217 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,219 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,219 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,233 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,233 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,246 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417f3a502a2a8bd456d93550a93a366a678 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f3a502a2a8bd456d93550a93a366a678 2023-04-17 03:55:12,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703772273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703772288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703772288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,333 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,343 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23cc2886[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,351 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,351 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,353 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,357 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,357 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,357 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,357 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,457 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,460 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,463 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,463 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,465 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,465 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,465 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,465 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@637852df[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,469 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,469 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,470 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,480 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,482 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,482 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,482 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,582 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,586 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,587 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,587 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,589 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,589 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,589 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,589 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,592 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@584c5ebe[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,592 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703772585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,595 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703772597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,606 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,606 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,609 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:12,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703772599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,614 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,614 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,615 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,615 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,652 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=266, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/3c8f8e10a7f54447bb34b8715d58597d 2023-04-17 03:55:12,716 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,719 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,722 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,722 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,722 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,722 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,724 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298f33b4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,729 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,730 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,732 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,733 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,733 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,733 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,733 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,738 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,739 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,839 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,842 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,845 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,848 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b45b61b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,852 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,856 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,856 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,860 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,860 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,860 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,865 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,865 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,865 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,965 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:12,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:12,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:12,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:12,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:12,969 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:12,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:12,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:12,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:12,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:12,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:12,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:12,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:12,973 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,973 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:12,973 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:12,973 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:12,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:12,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,975 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:12,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a8aef06[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:12,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:12,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:12,978 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,979 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:12,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:12,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,980 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:12,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:12,982 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:12,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:12,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:12,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:12,982 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:12,983 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,983 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,983 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:12,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:12,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:12,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,987 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:12,987 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:12,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:12,987 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:12,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:12,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:12,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:12,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:12,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:12,988 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:12,987 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,988 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:12,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:12,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:12,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:12,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/e3bfc9373e8a4ddfb63d7814e14d035f 2023-04-17 03:55:13,088 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:13,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:13,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:13,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:13,090 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:13,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:13,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:13,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:13,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:13,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:13,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:13,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:13,098 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:13,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:13,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:13,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:13,099 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,099 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:13,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7089e147[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:13,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:13,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:13,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703773094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:13,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:13,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:13,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:13,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:13,107 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:13,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:13,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:13,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:13,108 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:13,109 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,109 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,109 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,109 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,113 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,113 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:13,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,113 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703773106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:13,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703773111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,213 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:13,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:13,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:13,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:13,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:13,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:13,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:13,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:13,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:13,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:13,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:13,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:13,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:13,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:13,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:13,220 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:13,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:13,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:13,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204b6e71[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:13,224 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:13,224 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,225 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:13,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:13,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:13,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:13,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:13,227 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:13,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:13,227 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:13,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,229 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:13,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:13,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,234 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:13,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,234 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:13,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,334 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:13,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:13,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:13,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:13,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:13,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:13,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:13,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:13,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:13,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:13,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:13,341 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:13,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,342 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:13,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:13,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6678e10d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:13,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:13,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:13,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,352 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:13,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:13,354 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:13,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:13,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:13,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:13,354 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:13,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:13,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,360 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:13,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,360 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:13,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,460 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:13,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:13,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:13,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:13,463 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:13,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:13,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:13,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:13,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:13,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:13,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:13,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:13,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:13,467 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:13,467 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:13,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/d72b3a683a04467791f7d6d08d27389a 2023-04-17 03:55:13,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:13,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b17c01b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:13,474 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:13,474 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/3c8f8e10a7f54447bb34b8715d58597d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d 2023-04-17 03:55:13,476 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:13,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:13,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:13,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:13,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d, entries=200, sequenceid=266, filesize=32.3 K 2023-04-17 03:55:13,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/e3bfc9373e8a4ddfb63d7814e14d035f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f 2023-04-17 03:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:13,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:13,481 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:13,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:13,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:13,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:13,481 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:13,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:13,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:13,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f, entries=150, sequenceid=266, filesize=11.9 K 2023-04-17 03:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/d72b3a683a04467791f7d6d08d27389a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a 2023-04-17 03:55:13,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:13,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:13,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:13,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:13,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:13,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a, entries=150, sequenceid=266, filesize=11.9 K 2023-04-17 03:55:13,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:13,491 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:13,491 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:13,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=134.18 KB/137400 for ed08334fa02b7639086fbab0bf0f1d10 in 1659ms, sequenceid=266, compaction requested=true 2023-04-17 03:55:13,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:13,491 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:13,491 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:55:13,491 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:55:13,491 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:13,491 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:13,493 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155168 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:55:13,493 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/A is initiating minor compaction (all files) 2023-04-17 03:55:13,493 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/A in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,493 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=151.5 K 2023-04-17 03:55:13,494 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:55:13,494 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/B is initiating minor compaction (all files) 2023-04-17 03:55:13,494 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/B in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,494 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=67.9 K 2023-04-17 03:55:13,494 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e3ca79debf8c4056b4a7e4ebc83f4bee, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703706943 2023-04-17 03:55:13,494 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d926f6dc2eda4f7a837fce81c83703d5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703706943 2023-04-17 03:55:13,495 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0b7589ba21974eb5bc9c99a4615a6369, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681703707025 2023-04-17 03:55:13,495 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9400336a6e7c49a1ae51d1dc893d5289, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681703707025 2023-04-17 03:55:13,495 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5ce68b498d144ad4bec8334eeaae9c3a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681703707624 2023-04-17 03:55:13,495 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting adbfb1ad76044a929b0487af7e787ad2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681703707624 2023-04-17 03:55:13,495 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 79051e3f49224de9b752771cf6225207, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681703709675 2023-04-17 03:55:13,496 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9dc298fcec754156b27f8a30ff27a2ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681703709675 2023-04-17 03:55:13,496 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 16634477cec944bc9fb2161466adb5ef, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681703709999 2023-04-17 03:55:13,496 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 463d00d649664ed0ad2a2d52a9d46533, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681703709999 2023-04-17 03:55:13,496 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 3c8f8e10a7f54447bb34b8715d58597d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:13,497 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e3bfc9373e8a4ddfb63d7814e14d035f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:13,511 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#B#compaction#312 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:13,522 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#A#compaction#313 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:13,591 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:13,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:13,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:13,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:13,595 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:13,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:13,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:13,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,595 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,597 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,597 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:13,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:13,597 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:13,597 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:13,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:13,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:13,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:13,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:13,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:13,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:13,598 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,598 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:13,598 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:13,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:13,932 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/05fe0bf663694a9e9f08da34bd9d0630 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/05fe0bf663694a9e9f08da34bd9d0630 2023-04-17 03:55:13,936 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/B of ed08334fa02b7639086fbab0bf0f1d10 into 05fe0bf663694a9e9f08da34bd9d0630(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:13,937 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:13,937 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/B, priority=10, startTime=1681703713491; duration=0sec 2023-04-17 03:55:13,937 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:13,937 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-17 03:55:13,939 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-17 03:55:13,939 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/C is initiating minor compaction (all files) 2023-04-17 03:55:13,939 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/C in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:13,939 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=67.9 K 2023-04-17 03:55:13,939 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting bff919aaa830489a872bb89346fb4ddc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1681703706943 2023-04-17 03:55:13,940 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1e2c4d65904741a09ee93f2b37a1cce9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1681703707025 2023-04-17 03:55:13,940 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting a3aa5f6827054fcb9f17d0fd2fec3bba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681703707624 2023-04-17 03:55:13,940 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cfaf1126ad294bf7a938d166df34e33b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681703709675 2023-04-17 03:55:13,941 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c013206c74c64d7b8cfe5dc18e2c855f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1681703709999 2023-04-17 03:55:13,941 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d72b3a683a04467791f7d6d08d27389a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:13,950 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#C#compaction#315 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:13,960 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/acc7faac931b44dc929378733c9f4564 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/acc7faac931b44dc929378733c9f4564 2023-04-17 03:55:13,964 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/C of ed08334fa02b7639086fbab0bf0f1d10 into acc7faac931b44dc929378733c9f4564(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:13,964 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:13,964 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/C, priority=10, startTime=1681703713491; duration=0sec 2023-04-17 03:55:13,964 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:14,013 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171bcd7bb07cc3489e9e2971455ff3b305 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171bcd7bb07cc3489e9e2971455ff3b305 2023-04-17 03:55:14,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:14,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:14,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703774128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703774129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703774132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703774158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54556 deadline: 1681703774159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,168 DEBUG [Thread-2296] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:14,171 DEBUG [Thread-2300] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:14,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703774240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703774240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703774241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,357 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/700bdead73cb46d7bcc5a6ddec492ac4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/700bdead73cb46d7bcc5a6ddec492ac4 2023-04-17 03:55:14,362 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/A of ed08334fa02b7639086fbab0bf0f1d10 into 700bdead73cb46d7bcc5a6ddec492ac4(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:14,362 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:14,362 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/A, priority=10, startTime=1681703713491; duration=0sec 2023-04-17 03:55:14,362 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:14,418 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/2b9de47c589c457081247a69b2d404a4 2023-04-17 03:55:14,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703774444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703774445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703774448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703774749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703774753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:14,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703774754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:14,830 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/dc85aae83cb1441abbaa628c4747214e 2023-04-17 03:55:15,252 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/0b82605f98104bc7b06fce413ed27a86 2023-04-17 03:55:15,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/2b9de47c589c457081247a69b2d404a4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4 2023-04-17 03:55:15,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:15,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703775253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,259 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4, entries=150, sequenceid=301, filesize=25.4 K 2023-04-17 03:55:15,260 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/dc85aae83cb1441abbaa628c4747214e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e 2023-04-17 03:55:15,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:15,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703775259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:15,267 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e, entries=150, sequenceid=301, filesize=11.9 K 2023-04-17 03:55:15,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703775263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,268 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/0b82605f98104bc7b06fce413ed27a86 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86 2023-04-17 03:55:15,271 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86, entries=150, sequenceid=301, filesize=11.9 K 2023-04-17 03:55:15,272 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ed08334fa02b7639086fbab0bf0f1d10 in 1674ms, sequenceid=301, compaction requested=false 2023-04-17 03:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:15,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:15,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:15,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ab85e7[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:15,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:15,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:15,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:15,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:15,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:15,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:15,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:15,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:15,282 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:15,282 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:15,282 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:15,282 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:15,282 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:15,283 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,283 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,283 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:15,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:15,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:15,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:15,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:15,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:15,288 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:15,288 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:15,288 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:15,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:15,388 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:15,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:15,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:15,393 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:15,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:15,394 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,394 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:15,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:15,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:15,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:15,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:15,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:15,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:15,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:15,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:15,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:15,401 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:15,401 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:15,418 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041784f50968f1e34f73af9814ca7aea6a33 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041784f50968f1e34f73af9814ca7aea6a33 2023-04-17 03:55:15,824 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=314, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/a7981aa0e64c4ca4a843f5b41138aa4e 2023-04-17 03:55:15,832 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/5dafea6566e04b19a0acae090eb61cfc 2023-04-17 03:55:16,240 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/90c1e0ffe65443f3bdfac1ddfc7be519 2023-04-17 03:55:16,244 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/a7981aa0e64c4ca4a843f5b41138aa4e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e 2023-04-17 03:55:16,248 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e, entries=150, sequenceid=314, filesize=25.4 K 2023-04-17 03:55:16,251 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/5dafea6566e04b19a0acae090eb61cfc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc 2023-04-17 03:55:16,255 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc, entries=150, sequenceid=314, filesize=11.9 K 2023-04-17 03:55:16,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/90c1e0ffe65443f3bdfac1ddfc7be519 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519 2023-04-17 03:55:16,271 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519, entries=150, sequenceid=314, filesize=11.9 K 2023-04-17 03:55:16,272 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for ed08334fa02b7639086fbab0bf0f1d10 in 871ms, sequenceid=314, compaction requested=true 2023-04-17 03:55:16,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:16,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:16,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:16,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:16,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:16,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:16,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:16,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f6843e5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:16,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:16,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:16,284 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:16,285 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:16,290 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:16,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:16,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:16,294 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:16,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:16,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:16,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:16,294 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:16,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:16,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:16,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,299 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:16,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:16,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:16,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:16,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:16,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:16,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:16,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:16,301 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:16,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:16,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:16,303 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:16,303 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:16,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,403 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:16,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:16,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:16,408 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:16,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:16,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:16,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:16,408 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:16,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:16,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:16,410 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:16,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:16,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:16,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:16,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:16,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:16,412 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:16,412 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:16,412 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:16,413 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:16,413 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:16,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:16,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:16,442 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417afd55c6514044899a719dfde730791c8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417afd55c6514044899a719dfde730791c8 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703716448, max timestamp=1681703716468Num uniques 859; 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703716448, max timestamp=1681703716468Num uniques 859; 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703716448, max timestamp=1681703716468Num uniques 859; 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:16,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703776530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703776531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703776542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703776645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703776645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703776651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703776849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703776850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:16,858 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=323, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/8655490edcf24e52a8d3514846ed3e5d 2023-04-17 03:55:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703776860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703777158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703777158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703777168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,269 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/cb7d263a6247431b8df8aa439bba9dba 2023-04-17 03:55:17,284 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/378e28f333ea4821921b72520febd28f 2023-04-17 03:55:17,291 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/8655490edcf24e52a8d3514846ed3e5d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d 2023-04-17 03:55:17,294 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d, entries=200, sequenceid=323, filesize=32.3 K 2023-04-17 03:55:17,295 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/cb7d263a6247431b8df8aa439bba9dba as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba 2023-04-17 03:55:17,301 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba, entries=150, sequenceid=323, filesize=11.9 K 2023-04-17 03:55:17,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/378e28f333ea4821921b72520febd28f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f 2023-04-17 03:55:17,305 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f, entries=150, sequenceid=323, filesize=11.9 K 2023-04-17 03:55:17,306 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for ed08334fa02b7639086fbab0bf0f1d10 in 893ms, sequenceid=323, compaction requested=true 2023-04-17 03:55:17,306 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:17,306 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:17,306 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:17,306 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:17,306 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,307 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:17,307 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:17,308 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,308 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:17,309 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:17,309 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:17,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb9bd8c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:17,313 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:17,313 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:17,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:17,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:17,325 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:17,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:17,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:17,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:17,325 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:17,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:17,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:17,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,328 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:17,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:17,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:17,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:17,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:17,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,332 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:17,332 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:17,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:17,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,432 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:17,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:17,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:17,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:17,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:17,435 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:17,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:17,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:17,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:17,437 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:17,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:17,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:17,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:17,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:17,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:17,439 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:17,439 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:17,439 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-17 03:55:17,439 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:17,452 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417eebfc4534a7d45e6836357fdf15c7393 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eebfc4534a7d45e6836357fdf15c7393 2023-04-17 03:55:17,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:17,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:17,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703777680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703777683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703777684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703777789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703777790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703777792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,858 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/11b9fbb3008a44b1a39dcd9106ccfaeb 2023-04-17 03:55:17,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703777993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:17,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:17,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703777993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703777998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54556 deadline: 1681703778176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,182 DEBUG [Thread-2300] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:18,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54498 deadline: 1681703778206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,215 DEBUG [Thread-2296] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:18,279 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/f612578965c34f94aa784c49f6c04408 2023-04-17 03:55:18,290 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/e76bc9f27c3d4f2cb3b6b792f12387fa 2023-04-17 03:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/11b9fbb3008a44b1a39dcd9106ccfaeb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb 2023-04-17 03:55:18,302 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb, entries=150, sequenceid=359, filesize=25.4 K 2023-04-17 03:55:18,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/f612578965c34f94aa784c49f6c04408 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408 2023-04-17 03:55:18,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703778301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,306 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 03:55:18,306 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/e76bc9f27c3d4f2cb3b6b792f12387fa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa 2023-04-17 03:55:18,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703778301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703778304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,314 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa, entries=150, sequenceid=359, filesize=11.9 K 2023-04-17 03:55:18,315 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for ed08334fa02b7639086fbab0bf0f1d10 in 875ms, sequenceid=359, compaction requested=true 2023-04-17 03:55:18,315 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:18,315 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:18,315 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:18,320 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,320 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:18,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:18,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:18,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:18,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5beba158[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:18,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:18,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,323 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:18,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:18,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:18,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:18,326 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:18,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:18,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:18,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:18,328 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:18,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:18,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:18,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:18,328 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:18,330 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,330 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:18,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,330 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:18,330 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:18,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:18,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:18,336 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:18,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:18,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,336 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:18,336 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:18,336 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:18,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:18,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:18,436 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:18,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:18,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:18,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:18,440 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:18,440 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:18,440 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:18,440 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:18,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:18,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:18,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:18,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:18,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:18,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:18,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:18,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:18,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:18,446 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:55:18,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:18,447 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:18,447 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:18,447 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:18,447 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:18,447 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:18,468 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304170a8e60b479d74e60b9aa152c81fa62a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170a8e60b479d74e60b9aa152c81fa62a6 2023-04-17 03:55:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:18,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:18,882 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=368, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/5c742fed34d04f2386a2f4bd18e7995b 2023-04-17 03:55:18,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703718860, max timestamp=1681703718888Num uniques 859; 2023-04-17 03:55:18,890 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:18,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703718860, max timestamp=1681703718888Num uniques 859; 2023-04-17 03:55:18,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703718860, max timestamp=1681703718888Num uniques 859; 2023-04-17 03:55:18,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:18,891 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:18,898 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/f74e7cdefa4144a989427b8b99024d63 2023-04-17 03:55:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703778944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703778951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:18,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:18,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703778952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703779054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703779054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703779061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703779265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703779266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703779269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,312 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/cbff9a7b446f428cb395a40f497ff20e 2023-04-17 03:55:19,317 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/5c742fed34d04f2386a2f4bd18e7995b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b 2023-04-17 03:55:19,320 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b, entries=150, sequenceid=368, filesize=25.4 K 2023-04-17 03:55:19,321 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/f74e7cdefa4144a989427b8b99024d63 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63 2023-04-17 03:55:19,325 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 03:55:19,325 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/cbff9a7b446f428cb395a40f497ff20e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e 2023-04-17 03:55:19,328 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e, entries=150, sequenceid=368, filesize=11.9 K 2023-04-17 03:55:19,329 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for ed08334fa02b7639086fbab0bf0f1d10 in 883ms, sequenceid=368, compaction requested=true 2023-04-17 03:55:19,329 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:19,329 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:19,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:19,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:19,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:19,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43f7837f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:19,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:19,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,334 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:19,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:19,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:19,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:19,341 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:19,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:19,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:19,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:19,341 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:19,343 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,343 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,343 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:19,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:19,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:19,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:19,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:19,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:19,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:19,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:19,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:19,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:19,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:19,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:19,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:19,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,348 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:19,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:19,348 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:19,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,448 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:19,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:19,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:19,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:19,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:19,451 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:19,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:19,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:19,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:19,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:19,453 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,453 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:19,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:19,453 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,453 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:19,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:19,455 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:19,455 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:19,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:19,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703779595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703779594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703779594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54534 deadline: 1681703779706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703779706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703779706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,868 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417d906ea8402a345c9a1319201490dbd9b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d906ea8402a345c9a1319201490dbd9b 2023-04-17 03:55:19,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703779912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703779912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:19,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:19,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703779912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54510 deadline: 1681703780221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:54564 deadline: 1681703780221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703780222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,274 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=405, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/618526125a9e472e8243f59a6189c055 2023-04-17 03:55:20,687 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9dce5f50caa1415ea9ada8193dbfe612 2023-04-17 03:55:20,697 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/1dc4dbad5d8943ddabe0c692cfeda107 2023-04-17 03:55:20,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/618526125a9e472e8243f59a6189c055 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055 2023-04-17 03:55:20,703 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055, entries=150, sequenceid=405, filesize=25.4 K 2023-04-17 03:55:20,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9dce5f50caa1415ea9ada8193dbfe612 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612 2023-04-17 03:55:20,707 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 03:55:20,708 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/1dc4dbad5d8943ddabe0c692cfeda107 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107 2023-04-17 03:55:20,711 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107, entries=150, sequenceid=405, filesize=11.9 K 2023-04-17 03:55:20,712 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for ed08334fa02b7639086fbab0bf0f1d10 in 1257ms, sequenceid=405, compaction requested=true 2023-04-17 03:55:20,712 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:20,712 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:20,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:20,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3efd978f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:20,717 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:20,717 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:20,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:20,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:20,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:20,724 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:20,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:20,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:20,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:20,724 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:20,730 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,730 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,730 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,732 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:20,733 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:20,733 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:20,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:20,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:20,738 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:20,738 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:20,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703720758, max timestamp=1681703720808Num uniques 859; 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703720758, max timestamp=1681703720808Num uniques 859; 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703720758, max timestamp=1681703720808Num uniques 859; 2023-04-17 03:55:20,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:20,839 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:20,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:20,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:20,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:20,842 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,843 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:20,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,844 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:20,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:20,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:20,845 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:20,846 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:20,846 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:20,846 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:20,846 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:20,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:20,848 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:20,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dfb1801[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:20,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:20,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:20,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:20,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:20,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:20,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:20,855 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:20,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:20,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:20,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:20,855 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:20,857 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,857 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,861 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,861 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:20,861 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:20,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:20,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,862 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:20,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:20,862 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703780868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703780869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703780880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,962 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:20,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:20,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:20,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:20,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:20,967 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:20,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:20,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:20,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:20,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:20,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:20,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:20,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:20,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:20,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:20,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:20,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:20,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:20,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:20,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b9c9587[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:20,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:20,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,976 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:20,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:20,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:20,978 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:20,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:20,980 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:20,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:20,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:20,980 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:20,980 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,981 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,982 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:20,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:20,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:20,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:20,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:20,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:20,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:20,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:20,988 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:20,989 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:20,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:20,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703780987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703780987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:20,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:20,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703780991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,089 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,092 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,093 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,095 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,095 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,095 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,095 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,098 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@148f1ff0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,099 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,101 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,104 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,104 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,106 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,114 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,114 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,114 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,115 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,150 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417b4e8e1c372b246e297c5443cd9cbb831 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b4e8e1c372b246e297c5443cd9cbb831 2023-04-17 03:55:21,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703781197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703781197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703781198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,214 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,218 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,219 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,221 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,221 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,221 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,221 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,221 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,222 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@461cb301[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,226 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,226 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,229 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,232 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,234 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,236 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,238 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,239 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,239 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,239 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,339 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,348 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,349 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739280d3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,353 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,353 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,354 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,354 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,354 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,355 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,358 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,359 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,360 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,366 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,367 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,467 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,470 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cad138b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,489 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,492 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,492 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,495 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,496 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,499 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,499 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,498 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,499 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,499 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,499 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,499 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703781503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703781503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:21,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703781503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,556 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c0b8e9fcbca64074a6fc937989150cda 2023-04-17 03:55:21,599 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,602 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,603 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,607 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,612 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,612 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,615 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,617 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,617 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10b1f371[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,620 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,623 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,626 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,626 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,629 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,631 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,632 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,632 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,732 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,736 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,736 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,738 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,739 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,739 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,740 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,740 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,741 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,741 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,741 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,743 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3316cedd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,763 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,766 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,766 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,766 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,766 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,766 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,767 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,768 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,768 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,769 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,776 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,776 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,876 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:21,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:21,880 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:21,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:21,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:21,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,880 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,881 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,882 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,882 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:21,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:21,883 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,883 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:21,883 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:21,883 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:21,883 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:21,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54654474[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:21,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:21,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,888 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:21,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:21,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:21,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:21,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:21,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:21,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:21,894 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:21,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:21,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:21,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:21,894 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:21,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,895 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,896 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:21,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:21,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:21,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:21,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:21,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:21,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:21,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:21,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:21,901 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:21,901 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:21,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/56ad2564254d4d1eb841f13e44716c91 2023-04-17 03:55:21,983 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/35119223318d40aa93fc3934ad515d2a 2023-04-17 03:55:21,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c0b8e9fcbca64074a6fc937989150cda as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda 2023-04-17 03:55:21,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda, entries=150, sequenceid=415, filesize=25.4 K 2023-04-17 03:55:21,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/56ad2564254d4d1eb841f13e44716c91 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91 2023-04-17 03:55:21,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 03:55:21,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/35119223318d40aa93fc3934ad515d2a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a 2023-04-17 03:55:21,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 03:55:21,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ed08334fa02b7639086fbab0bf0f1d10 in 1265ms, sequenceid=415, compaction requested=true 2023-04-17 03:55:21,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:21,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:21,999 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-17 03:55:21,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:21,999 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-17 03:55:21,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:22,001 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97694 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-17 03:55:22,001 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 215614 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-17 03:55:22,001 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/B is initiating minor compaction (all files) 2023-04-17 03:55:22,001 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/A is initiating minor compaction (all files) 2023-04-17 03:55:22,002 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/B in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,002 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/A in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,002 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/05fe0bf663694a9e9f08da34bd9d0630, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=95.4 K 2023-04-17 03:55:22,002 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/700bdead73cb46d7bcc5a6ddec492ac4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=210.6 K 2023-04-17 03:55:22,002 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 05fe0bf663694a9e9f08da34bd9d0630, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:22,002 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 700bdead73cb46d7bcc5a6ddec492ac4, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:22,003 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting dc85aae83cb1441abbaa628c4747214e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681703711951 2023-04-17 03:55:22,003 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,003 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2b9de47c589c457081247a69b2d404a4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681703711951 2023-04-17 03:55:22,003 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5dafea6566e04b19a0acae090eb61cfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703714126 2023-04-17 03:55:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,004 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a7981aa0e64c4ca4a843f5b41138aa4e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703714126 2023-04-17 03:55:22,004 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cb7d263a6247431b8df8aa439bba9dba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681703716405 2023-04-17 03:55:22,004 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8655490edcf24e52a8d3514846ed3e5d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681703716340 2023-04-17 03:55:22,005 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f612578965c34f94aa784c49f6c04408, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681703716492 2023-04-17 03:55:22,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,005 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 11b9fbb3008a44b1a39dcd9106ccfaeb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681703716492 2023-04-17 03:55:22,006 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f74e7cdefa4144a989427b8b99024d63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681703717679 2023-04-17 03:55:22,006 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9dce5f50caa1415ea9ada8193dbfe612, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681703718918 2023-04-17 03:55:22,006 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5c742fed34d04f2386a2f4bd18e7995b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681703717679 2023-04-17 03:55:22,007 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 56ad2564254d4d1eb841f13e44716c91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:22,008 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 618526125a9e472e8243f59a6189c055, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681703718918 2023-04-17 03:55:22,008 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c0b8e9fcbca64074a6fc937989150cda, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,021 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,024 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,024 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,025 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,026 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,026 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:22,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 03:55:22,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703720853, max timestamp=1681703722030Num uniques 1068; 2023-04-17 03:55:22,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 03:55:22,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703720853, max timestamp=1681703722030Num uniques 1068; 2023-04-17 03:55:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,033 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 03:55:22,033 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,033 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,033 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,033 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,034 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703720853, max timestamp=1681703722030Num uniques 1068; 2023-04-17 03:55:22,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:22,036 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,043 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#B#compaction#336 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:22,043 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,048 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40402438[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,051 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,051 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,052 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,052 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,058 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,058 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,058 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,060 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#A#compaction#337 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:22,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,061 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,062 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,066 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,067 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,067 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,067 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:22,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703722104, max timestamp=1681703722127Num uniques 859; 2023-04-17 03:55:22,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:22,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703722104, max timestamp=1681703722127Num uniques 859; 2023-04-17 03:55:22,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:22,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703722104, max timestamp=1681703722127Num uniques 859; 2023-04-17 03:55:22,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:22,167 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,171 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,171 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,171 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,171 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,172 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,172 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,173 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,173 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,182 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,182 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff56d3a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,188 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,188 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,190 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,190 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,190 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,203 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,203 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,204 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,204 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,204 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703722182, max timestamp=1681703722208Num uniques 1018; 2023-04-17 03:55:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:22,210 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703722182, max timestamp=1681703722208Num uniques 1018; 2023-04-17 03:55:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:22,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703722182, max timestamp=1681703722208Num uniques 1018; 2023-04-17 03:55:22,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:22,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,213 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,213 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,213 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,213 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:22,213 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,214 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,214 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,214 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703782297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703722265, max timestamp=1681703722311Num uniques 1077; 2023-04-17 03:55:22,314 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:22,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703722265, max timestamp=1681703722311Num uniques 1077; 2023-04-17 03:55:22,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:22,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703722265, max timestamp=1681703722311Num uniques 1077; 2023-04-17 03:55:22,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,320 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,320 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,320 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,320 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,321 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,321 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,323 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,323 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,324 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,324 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,324 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,325 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a10203f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,329 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,334 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,334 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,341 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,342 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703722390, max timestamp=1681703722422Num uniques 1036; 2023-04-17 03:55:22,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-17 03:55:22,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703722390, max timestamp=1681703722422Num uniques 1036; 2023-04-17 03:55:22,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-17 03:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703722390, max timestamp=1681703722422Num uniques 1036; 2023-04-17 03:55:22,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-17 03:55:22,442 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,445 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,447 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,447 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,452 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,456 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a899ed[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,468 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,472 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,472 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,474 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,474 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,479 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/c712dc8e80cf44bca83d663610050b18 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/c712dc8e80cf44bca83d663610050b18 2023-04-17 03:55:22,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,486 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/B of ed08334fa02b7639086fbab0bf0f1d10 into c712dc8e80cf44bca83d663610050b18(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:22,486 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:22,486 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/B, priority=8, startTime=1681703721999; duration=0sec 2023-04-17 03:55:22,486 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:22,486 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-17 03:55:22,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,487 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,487 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,489 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417019bed7c1db04faf97395d9c25c1ee16 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417019bed7c1db04faf97395d9c25c1ee16 2023-04-17 03:55:22,489 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97694 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-17 03:55:22,489 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/C is initiating minor compaction (all files) 2023-04-17 03:55:22,489 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/C in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,489 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/acc7faac931b44dc929378733c9f4564, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=95.4 K 2023-04-17 03:55:22,490 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting acc7faac931b44dc929378733c9f4564, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681703711817 2023-04-17 03:55:22,490 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0b82605f98104bc7b06fce413ed27a86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681703711951 2023-04-17 03:55:22,490 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 90c1e0ffe65443f3bdfac1ddfc7be519, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1681703714126 2023-04-17 03:55:22,491 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 378e28f333ea4821921b72520febd28f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681703716405 2023-04-17 03:55:22,492 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e76bc9f27c3d4f2cb3b6b792f12387fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681703716492 2023-04-17 03:55:22,492 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cbff9a7b446f428cb395a40f497ff20e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681703717679 2023-04-17 03:55:22,492 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1dc4dbad5d8943ddabe0c692cfeda107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681703718918 2023-04-17 03:55:22,493 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 35119223318d40aa93fc3934ad515d2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:22,498 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=453, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/dc1cdf1597544889b8ce36d10fcc6835 2023-04-17 03:55:22,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703722479, max timestamp=1681703722505Num uniques 1095; 2023-04-17 03:55:22,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703722479, max timestamp=1681703722505Num uniques 1095; 2023-04-17 03:55:22,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:22,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:22,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703722479, max timestamp=1681703722505Num uniques 1095; 2023-04-17 03:55:22,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:22,518 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#C#compaction#340 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:22,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/4b2e0b170b484e559ff1943fc487ec12 2023-04-17 03:55:22,529 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/d0a6f9d5bdb2468d90a73842288c6d9c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d0a6f9d5bdb2468d90a73842288c6d9c 2023-04-17 03:55:22,533 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/C of ed08334fa02b7639086fbab0bf0f1d10 into d0a6f9d5bdb2468d90a73842288c6d9c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:22,533 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:22,533 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/C, priority=8, startTime=1681703721999; duration=0sec 2023-04-17 03:55:22,533 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:22,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703782572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703782572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703782575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,587 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,590 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,593 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,597 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,598 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650818a4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,600 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,600 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,600 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,600 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,604 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,604 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,604 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,604 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,604 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,607 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,610 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,613 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,613 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,613 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,613 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,613 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,613 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703782681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703782681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703782681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,713 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,716 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,719 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,720 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,720 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,720 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11602582[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,723 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,723 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,725 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,725 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,725 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,727 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,729 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,729 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,732 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,734 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,735 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,735 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,735 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:22,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,835 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,838 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,846 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,847 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:22,848 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:22,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:22,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:22,848 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:22,849 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a2aee49[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:22,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:22,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,852 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:22,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:22,856 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:22,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:22,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:22,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:22,856 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:22,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:22,866 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:22,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:22,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:22,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:22,870 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:22,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,871 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:22,871 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:22,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:22,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:22,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:22,871 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703782886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703782887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703782887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/ae8dac417f7f4cbd9569021607580c81 2023-04-17 03:55:22,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/dc1cdf1597544889b8ce36d10fcc6835 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835 2023-04-17 03:55:22,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835, entries=250, sequenceid=453, filesize=39.2 K 2023-04-17 03:55:22,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/4b2e0b170b484e559ff1943fc487ec12 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12 2023-04-17 03:55:22,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12, entries=150, sequenceid=453, filesize=11.9 K 2023-04-17 03:55:22,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/ae8dac417f7f4cbd9569021607580c81 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81 2023-04-17 03:55:22,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81, entries=150, sequenceid=453, filesize=11.9 K 2023-04-17 03:55:22,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=154.31 KB/158010 for ed08334fa02b7639086fbab0bf0f1d10 in 935ms, sequenceid=453, compaction requested=false 2023-04-17 03:55:22,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:22,971 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:22,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:22,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:22,974 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:22,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:22,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:22,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:22,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:22,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:22,977 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:22,977 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:22,977 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:22,990 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041765381b26b2fc44f6a9c09aeabd49c97e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765381b26b2fc44f6a9c09aeabd49c97e 2023-04-17 03:55:23,008 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=561, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/f2847cfa19f14e6aaaa3a27490d5bc1f 2023-04-17 03:55:23,027 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/20bf1ebdb0164e76808adce9da05f0bf 2023-04-17 03:55:23,046 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/b0fc441d0bf7486884312436e1bef962 2023-04-17 03:55:23,050 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/f2847cfa19f14e6aaaa3a27490d5bc1f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f 2023-04-17 03:55:23,055 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f, entries=150, sequenceid=561, filesize=25.4 K 2023-04-17 03:55:23,056 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/20bf1ebdb0164e76808adce9da05f0bf as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf 2023-04-17 03:55:23,059 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf, entries=150, sequenceid=561, filesize=11.9 K 2023-04-17 03:55:23,063 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/b0fc441d0bf7486884312436e1bef962 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962 2023-04-17 03:55:23,098 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962, entries=150, sequenceid=561, filesize=11.9 K 2023-04-17 03:55:23,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=0 B/0 for ed08334fa02b7639086fbab0bf0f1d10 in 126ms, sequenceid=561, compaction requested=true 2023-04-17 03:55:23,104 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:23,104 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:23,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,106 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:23,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2d7110[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:23,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:23,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,110 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:23,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:23,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:23,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:23,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:23,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:23,115 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:23,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:23,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:23,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:23,115 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:23,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,121 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:23,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:23,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:23,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,121 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:23,121 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:23,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,122 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,221 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:23,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:23,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:23,225 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:23,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:23,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:23,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,228 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:23,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:23,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:23,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:23,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:23,232 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,232 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:23,232 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-17 03:55:23,232 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:23,232 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:23,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:23,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:23,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:23,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:23,253 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041793014d92ee4146689a0dd6f916caf522 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041793014d92ee4146689a0dd6f916caf522 2023-04-17 03:55:23,297 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/833c5b9b1dff44efa3010b23ea90e05a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/833c5b9b1dff44efa3010b23ea90e05a 2023-04-17 03:55:23,301 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/A of ed08334fa02b7639086fbab0bf0f1d10 into 833c5b9b1dff44efa3010b23ea90e05a(size=25.9 K), total size for store is 90.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 03:55:23,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:23,301 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/A, priority=8, startTime=1681703721998; duration=1sec 2023-04-17 03:55:23,301 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:23,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:23,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723347, max timestamp=1681703723361Num uniques 859; 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723347, max timestamp=1681703723361Num uniques 859; 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723347, max timestamp=1681703723361Num uniques 859; 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703723380, max timestamp=1681703723412Num uniques 1118; 2023-04-17 03:55:23,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703723380, max timestamp=1681703723412Num uniques 1118; 2023-04-17 03:55:23,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:23,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:23,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703723380, max timestamp=1681703723412Num uniques 1118; 2023-04-17 03:55:23,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:23,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703723435, max timestamp=1681703723449Num uniques 1077; 2023-04-17 03:55:23,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:23,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703723435, max timestamp=1681703723449Num uniques 1077; 2023-04-17 03:55:23,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:23,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681703723435, max timestamp=1681703723449Num uniques 1077; 2023-04-17 03:55:23,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-17 03:55:23,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681703723449, max timestamp=1681703723486Num uniques 1086; 2023-04-17 03:55:23,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-17 03:55:23,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681703723449, max timestamp=1681703723486Num uniques 1086; 2023-04-17 03:55:23,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-17 03:55:23,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=386, cellSize=17676, totalHeapSize=40384, min timestamp=1681703723449, max timestamp=1681703723486Num uniques 1086; 2023-04-17 03:55:23,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=17676, suffix heap size=140912,new segment heap size=40384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=386 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681703723513, max timestamp=1681703723521Num uniques 1245; 2023-04-17 03:55:23,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-17 03:55:23,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703783522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681703723513, max timestamp=1681703723521Num uniques 1245; 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681703723513, max timestamp=1681703723521Num uniques 1245; 2023-04-17 03:55:23,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=57020, new segment data size=11220, suffix heap size=157448,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1245, new segment cells count=245 2023-04-17 03:55:23,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703723554, max timestamp=1681703723575Num uniques 1104; 2023-04-17 03:55:23,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703723554, max timestamp=1681703723575Num uniques 1104; 2023-04-17 03:55:23,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-17 03:55:23,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681703723554, max timestamp=1681703723575Num uniques 1104; 2023-04-17 03:55:23,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-17 03:55:23,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-17 03:55:23,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703723606, max timestamp=1681703723636Num uniques 1013; 2023-04-17 03:55:23,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-17 03:55:23,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703723606, max timestamp=1681703723638Num uniques 1013; 2023-04-17 03:55:23,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703723606, max timestamp=1681703723638Num uniques 1013; 2023-04-17 03:55:23,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-17 03:55:23,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-17 03:55:23,671 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=566, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/548a602cb4e54ca29ef6e58b5e4d4ced 2023-04-17 03:55:23,680 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/d015377bfcba4851b4cb9ad0861ac573 2023-04-17 03:55:23,688 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=566 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/303e1fadd82043d2bb8dbe0ebf7921f0 2023-04-17 03:55:23,694 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/548a602cb4e54ca29ef6e58b5e4d4ced as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced 2023-04-17 03:55:23,703 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced, entries=100, sequenceid=566, filesize=18.6 K 2023-04-17 03:55:23,703 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/d015377bfcba4851b4cb9ad0861ac573 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573 2023-04-17 03:55:23,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703723685, max timestamp=1681703723703Num uniques 1022; 2023-04-17 03:55:23,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-17 03:55:23,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703723685, max timestamp=1681703723703Num uniques 1022; 2023-04-17 03:55:23,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-17 03:55:23,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1681703723685, max timestamp=1681703723703Num uniques 1022; 2023-04-17 03:55:23,706 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-04-17 03:55:23,709 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573, entries=50, sequenceid=566, filesize=7.1 K 2023-04-17 03:55:23,710 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/303e1fadd82043d2bb8dbe0ebf7921f0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0 2023-04-17 03:55:23,714 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0, entries=50, sequenceid=566, filesize=7.1 K 2023-04-17 03:55:23,715 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=53.67 KB/54960 for ed08334fa02b7639086fbab0bf0f1d10 in 483ms, sequenceid=566, compaction requested=true 2023-04-17 03:55:23,715 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:23,715 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:23,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:23,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=60.38 KB heapSize=155.25 KB 2023-04-17 03:55:23,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:23,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:23,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:23,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:23,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:23,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:23,717 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f4581c9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:23,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:23,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,720 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:23,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,720 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:23,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,722 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:23,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:23,724 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:23,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:23,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:23,724 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:23,724 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:23,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,729 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,733 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:23,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:23,733 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,734 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:23,734 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:23,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723746, max timestamp=1681703723772Num uniques 859; 2023-04-17 03:55:23,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723746, max timestamp=1681703723772Num uniques 859; 2023-04-17 03:55:23,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703723746, max timestamp=1681703723772Num uniques 859; 2023-04-17 03:55:23,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:23,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703783805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703783808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703783815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,834 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:23,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:23,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:23,837 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:23,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:23,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:23,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:23,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:23,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:23,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:23,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f19e996[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:23,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:23,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:23,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:23,849 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,849 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:23,851 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:23,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:23,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:23,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:23,852 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:23,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,853 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:23,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,863 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:23,863 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:23,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703783917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703783918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:23,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703783923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,964 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:23,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:23,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:23,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:23,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:23,967 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:23,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:23,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:23,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,968 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,969 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:23,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:23,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:23,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:23,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:23,971 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:23,971 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:23,972 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:23,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c049a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:23,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:23,974 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:23,976 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:23,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:23,979 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:23,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:23,982 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:23,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:23,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:23,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:23,982 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:23,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:23,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:23,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:23,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:23,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:23,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:23,988 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:23,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:23,989 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:23,989 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:23,988 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:23,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:24,089 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:24,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:24,092 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:24,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:24,092 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:24,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:24,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:24,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:24,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:24,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:24,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:24,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:24,097 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:24,097 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:24,098 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:24,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:24,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43b84030[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:24,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:24,102 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,103 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:24,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:24,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,103 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:24,104 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,105 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:24,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:24,107 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:24,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:24,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:24,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:24,107 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:24,108 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,108 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,108 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,110 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,111 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,117 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:24,117 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,117 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703784122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703784128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,137 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703784130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,142 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172a1a59ffd2644833b37619e03ab1ce76 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172a1a59ffd2644833b37619e03ab1ce76 2023-04-17 03:55:24,147 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=711, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/70edae6c57f74ad8bf076a2b6e3192e5 2023-04-17 03:55:24,217 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:24,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:24,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:24,220 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:24,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:24,220 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:24,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,224 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,225 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:24,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:24,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:24,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:24,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,227 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:24,228 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,228 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:24,228 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:24,228 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:24,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:24,230 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:24,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6050da47[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:24,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:24,232 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:24,233 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:24,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:24,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:24,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,235 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:24,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:24,239 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:24,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:24,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:24,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:24,239 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:24,241 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,241 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,241 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,242 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:24,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,249 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:24,249 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:24,249 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,350 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:24,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:24,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:24,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:24,360 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:24,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:24,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:24,360 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,361 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,363 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:24,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:24,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:24,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:24,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:24,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:24,367 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,367 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:24,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:24,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:24,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:24,369 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,370 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,370 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,370 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:24,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3055da82[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:24,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:24,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,374 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:24,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:24,374 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:24,374 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,374 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:24,375 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:24,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:24,379 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:24,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:24,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:24,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:24,379 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:24,386 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,386 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,386 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,387 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:24,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:24,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,393 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:24,393 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:24,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703784429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703784434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703784440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,493 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:24,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:24,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:24,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:24,504 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:24,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:24,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:24,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,505 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,506 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:24,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:24,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:24,520 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:24,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,522 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:24,523 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:24,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524f70c6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:24,523 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,528 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,528 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:24,528 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,528 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:24,528 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:24,528 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:24,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,529 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,530 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:24,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:24,532 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:24,532 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:24,532 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:24,532 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:24,532 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:24,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,535 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:24,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:24,538 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:24,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:24,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:24,538 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,540 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:24,540 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:24,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:24,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:24,557 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=711 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/374db406d6de49ac8b5d9fa95d70443d 2023-04-17 03:55:24,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=711 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/7a07ad9e5e674185bb13624ab10dfcf1 2023-04-17 03:55:24,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/70edae6c57f74ad8bf076a2b6e3192e5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5 2023-04-17 03:55:24,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5, entries=250, sequenceid=711, filesize=39.2 K 2023-04-17 03:55:24,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/374db406d6de49ac8b5d9fa95d70443d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d 2023-04-17 03:55:24,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d, entries=150, sequenceid=711, filesize=11.9 K 2023-04-17 03:55:24,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/7a07ad9e5e674185bb13624ab10dfcf1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1 2023-04-17 03:55:24,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1, entries=150, sequenceid=711, filesize=11.9 K 2023-04-17 03:55:24,609 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~155.20 KB/158928, currentSize=134.18 KB/137400 for ed08334fa02b7639086fbab0bf0f1d10 in 894ms, sequenceid=711, compaction requested=true 2023-04-17 03:55:24,609 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:24,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:24,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:24,609 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:24,609 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:24,609 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:24,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:24,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/B is initiating minor compaction (all files) 2023-04-17 03:55:24,611 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/B in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,611 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/c712dc8e80cf44bca83d663610050b18, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=55.2 K 2023-04-17 03:55:24,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c712dc8e80cf44bca83d663610050b18, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:24,612 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 151826 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:24,612 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/A is initiating minor compaction (all files) 2023-04-17 03:55:24,612 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/A in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,612 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/833c5b9b1dff44efa3010b23ea90e05a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=148.3 K 2023-04-17 03:55:24,612 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4b2e0b170b484e559ff1943fc487ec12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703720853 2023-04-17 03:55:24,612 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 833c5b9b1dff44efa3010b23ea90e05a, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 20bf1ebdb0164e76808adce9da05f0bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681703722558 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting dc1cdf1597544889b8ce36d10fcc6835, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703720853 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d015377bfcba4851b4cb9ad0861ac573, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681703723231 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f2847cfa19f14e6aaaa3a27490d5bc1f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681703722558 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 374db406d6de49ac8b5d9fa95d70443d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=711, earliestPutTs=1681703723698 2023-04-17 03:55:24,613 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 548a602cb4e54ca29ef6e58b5e4d4ced, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681703723230 2023-04-17 03:55:24,614 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 70edae6c57f74ad8bf076a2b6e3192e5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=711, earliestPutTs=1681703723685 2023-04-17 03:55:24,623 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#B#compaction#351 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:24,630 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#A#compaction#352 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:24,640 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:24,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:24,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:24,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:24,644 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:24,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:24,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:24,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,646 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:24,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:24,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:24,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:24,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:24,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:24,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:24,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:24,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:24,648 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:24,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:24,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:24,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703784962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703784963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:24,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703784964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,037 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/7d8d636607cf4659918673519cefc2ba as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7d8d636607cf4659918673519cefc2ba 2023-04-17 03:55:25,041 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/B of ed08334fa02b7639086fbab0bf0f1d10 into 7d8d636607cf4659918673519cefc2ba(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:25,041 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:25,041 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/B, priority=11, startTime=1681703724609; duration=0sec 2023-04-17 03:55:25,042 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:25,042 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:25,044 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:25,044 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): ed08334fa02b7639086fbab0bf0f1d10/C is initiating minor compaction (all files) 2023-04-17 03:55:25,044 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ed08334fa02b7639086fbab0bf0f1d10/C in TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:25,044 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d0a6f9d5bdb2468d90a73842288c6d9c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp, totalSize=55.2 K 2023-04-17 03:55:25,044 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d0a6f9d5bdb2468d90a73842288c6d9c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703719593 2023-04-17 03:55:25,045 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ae8dac417f7f4cbd9569021607580c81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703720853 2023-04-17 03:55:25,045 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b0fc441d0bf7486884312436e1bef962, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681703722558 2023-04-17 03:55:25,046 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 303e1fadd82043d2bb8dbe0ebf7921f0, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=566, earliestPutTs=1681703723231 2023-04-17 03:55:25,046 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7a07ad9e5e674185bb13624ab10dfcf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=711, earliestPutTs=1681703723698 2023-04-17 03:55:25,055 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): ed08334fa02b7639086fbab0bf0f1d10#C#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:25,066 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/594b4b696cfb4e838c5d22baa8b43730 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/594b4b696cfb4e838c5d22baa8b43730 2023-04-17 03:55:25,067 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173c7d8b35273f4590a29ed93dc896dba4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173c7d8b35273f4590a29ed93dc896dba4 2023-04-17 03:55:25,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703785073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,073 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/C of ed08334fa02b7639086fbab0bf0f1d10 into 594b4b696cfb4e838c5d22baa8b43730(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:25,074 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:25,074 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/C, priority=11, startTime=1681703724609; duration=0sec 2023-04-17 03:55:25,074 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:25,082 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=746, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/6147654da0754a719ea7ae15a0407d15 2023-04-17 03:55:25,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703785075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703785078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703785276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703785298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703785298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,453 WARN [DataStreamer for file /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c9f7756c00e14d24b2d89fe0e1a161b3] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-17 03:55:25,458 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c9f7756c00e14d24b2d89fe0e1a161b3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c9f7756c00e14d24b2d89fe0e1a161b3 2023-04-17 03:55:25,463 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ed08334fa02b7639086fbab0bf0f1d10/A of ed08334fa02b7639086fbab0bf0f1d10 into c9f7756c00e14d24b2d89fe0e1a161b3(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:25,463 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:25,463 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10., storeName=ed08334fa02b7639086fbab0bf0f1d10/A, priority=11, startTime=1681703724609; duration=0sec 2023-04-17 03:55:25,464 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:25,518 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/52787a167d594a6599e009b9020ab3c3 2023-04-17 03:55:25,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703785584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703785605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:25,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703785606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,928 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/6b745ba4315a42a9ab04dd549ede5eff 2023-04-17 03:55:25,932 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/6147654da0754a719ea7ae15a0407d15 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/6147654da0754a719ea7ae15a0407d15 2023-04-17 03:55:25,935 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/6147654da0754a719ea7ae15a0407d15, entries=150, sequenceid=746, filesize=25.4 K 2023-04-17 03:55:25,936 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/52787a167d594a6599e009b9020ab3c3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/52787a167d594a6599e009b9020ab3c3 2023-04-17 03:55:25,941 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/52787a167d594a6599e009b9020ab3c3, entries=150, sequenceid=746, filesize=11.9 K 2023-04-17 03:55:25,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/6b745ba4315a42a9ab04dd549ede5eff as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6b745ba4315a42a9ab04dd549ede5eff 2023-04-17 03:55:25,946 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6b745ba4315a42a9ab04dd549ede5eff, entries=150, sequenceid=746, filesize=11.9 K 2023-04-17 03:55:25,947 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ed08334fa02b7639086fbab0bf0f1d10 in 1299ms, sequenceid=746, compaction requested=false 2023-04-17 03:55:25,947 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:25,947 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:25,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:25,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:25,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:25,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:25,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f7037f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:25,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:25,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,953 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:25,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:25,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:25,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:25,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:25,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:25,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:25,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:25,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:25,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:25,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:25,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:25,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:25,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,963 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:25,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:25,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:25,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:25,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:25,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:25,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:25,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:25,967 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:25,967 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:26,068 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:26,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:26,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:26,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:26,071 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:26,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:26,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:26,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:26,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:26,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:26,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:26,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:26,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:26,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:26,075 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:26,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:26,092 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417b5b2f64bce244e598d6ffe3666b035bb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b5b2f64bce244e598d6ffe3666b035bb 2023-04-17 03:55:26,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:26,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:26,100 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=759, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/e78ad240801a49fcb36b8412e4a0b6c6 2023-04-17 03:55:26,110 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/7327801c0d484235adfd0e151822a4e7 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703726128, max timestamp=1681703726141Num uniques 859; 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703726128, max timestamp=1681703726141Num uniques 859; 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703726128, max timestamp=1681703726141Num uniques 859; 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:26,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:26,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703786175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703786176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703786177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703786279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703786285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703786286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703786484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703786489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703786489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,518 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a155d48e65234cba88818cab354e72f7 2023-04-17 03:55:26,521 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/e78ad240801a49fcb36b8412e4a0b6c6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e78ad240801a49fcb36b8412e4a0b6c6 2023-04-17 03:55:26,524 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e78ad240801a49fcb36b8412e4a0b6c6, entries=150, sequenceid=759, filesize=25.4 K 2023-04-17 03:55:26,525 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/7327801c0d484235adfd0e151822a4e7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7327801c0d484235adfd0e151822a4e7 2023-04-17 03:55:26,528 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7327801c0d484235adfd0e151822a4e7, entries=150, sequenceid=759, filesize=11.9 K 2023-04-17 03:55:26,528 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a155d48e65234cba88818cab354e72f7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a155d48e65234cba88818cab354e72f7 2023-04-17 03:55:26,531 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a155d48e65234cba88818cab354e72f7, entries=150, sequenceid=759, filesize=11.9 K 2023-04-17 03:55:26,532 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for ed08334fa02b7639086fbab0bf0f1d10 in 457ms, sequenceid=759, compaction requested=true 2023-04-17 03:55:26,532 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:26,532 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:26,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:26,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:26,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:26,540 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:26,541 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,541 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:26,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@359369a4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:26,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:26,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,543 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:26,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:26,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:26,543 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:26,544 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,544 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:26,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:26,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:26,547 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:26,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:26,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:26,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:26,547 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:26,549 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:26,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:26,549 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:26,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,553 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:26,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:26,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:26,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:26,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,554 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:26,554 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:26,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:26,555 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:26,555 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:26,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:26,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:26,654 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:26,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:26,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:26,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:26,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:26,659 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:26,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:26,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:26,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:26,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,659 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,661 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:26,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:26,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:26,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:26,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:26,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:26,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:26,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:26,677 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:26,677 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. started... 2023-04-17 03:55:26,684 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:26,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:26,685 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:26,685 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:26,710 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e3b2a27055f64e4698deee4283e6f9a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3b2a27055f64e4698deee4283e6f9a6 2023-04-17 03:55:26,716 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=795, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/077865f8e38a43bbb9953f16b53cb797 2023-04-17 03:55:26,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:26,793 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. as already flushing 2023-04-17 03:55:26,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703786810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703786811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703786811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,922 DEBUG [Thread-2311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ce79633 to 127.0.0.1:60620 2023-04-17 03:55:26,922 DEBUG [Thread-2311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:26,928 DEBUG [Thread-2307] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:60620 2023-04-17 03:55:26,928 DEBUG [Thread-2307] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:26,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703786919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703786919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:26,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703786920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:26,930 DEBUG [Thread-2305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:60620 2023-04-17 03:55:26,930 DEBUG [Thread-2305] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:26,931 DEBUG [Thread-2309] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:60620 2023-04-17 03:55:26,931 DEBUG [Thread-2309] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:26,935 DEBUG [Thread-2303] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:60620 2023-04-17 03:55:26,935 DEBUG [Thread-2303] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:27,127 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/763791935aa544faba8b205c67bc295d 2023-04-17 03:55:27,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703787130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703787131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703787132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54510 deadline: 1681703787433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54534 deadline: 1681703787434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:27,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:54564 deadline: 1681703787435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,536 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/75b33601bfee41179bc65e7179803cb6 2023-04-17 03:55:27,539 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/077865f8e38a43bbb9953f16b53cb797 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/077865f8e38a43bbb9953f16b53cb797 2023-04-17 03:55:27,542 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/077865f8e38a43bbb9953f16b53cb797, entries=150, sequenceid=795, filesize=25.4 K 2023-04-17 03:55:27,543 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/763791935aa544faba8b205c67bc295d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/763791935aa544faba8b205c67bc295d 2023-04-17 03:55:27,547 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/763791935aa544faba8b205c67bc295d, entries=150, sequenceid=795, filesize=11.9 K 2023-04-17 03:55:27,547 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/75b33601bfee41179bc65e7179803cb6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/75b33601bfee41179bc65e7179803cb6 2023-04-17 03:55:27,551 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/75b33601bfee41179bc65e7179803cb6, entries=150, sequenceid=795, filesize=11.9 K 2023-04-17 03:55:27,551 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for ed08334fa02b7639086fbab0bf0f1d10 in 867ms, sequenceid=795, compaction requested=true 2023-04-17 03:55:27,551 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:27,552 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:27,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:27,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b8c345e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:27,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:27,554 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,555 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,555 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:27,556 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,556 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:27,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:27,557 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:27,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:27,558 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:27,559 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:27,559 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:27,559 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:27,559 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:27,561 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,561 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,561 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:27,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:27,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:27,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:27,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:27,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:27,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:27,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:27,565 DEBUG [Thread-2302] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:27,565 DEBUG [Thread-2302] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:27,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:27,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:27,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:27,665 DEBUG [Thread-2302] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:27,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:27,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:27,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:55:27,938 DEBUG [Thread-2292] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:60620 2023-04-17 03:55:27,938 DEBUG [Thread-2292] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:27,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:27,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:27,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:27,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:27,939 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:27,939 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:27,939 DEBUG [Thread-2294] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:60620 2023-04-17 03:55:27,939 DEBUG [Thread-2294] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:27,943 DEBUG [Thread-2298] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:60620 2023-04-17 03:55:27,943 DEBUG [Thread-2298] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:27,953 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e3cb2921aa0c428997a5c2f90dd45cc7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3cb2921aa0c428997a5c2f90dd45cc7 2023-04-17 03:55:28,260 DEBUG [Thread-2300] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:60620 2023-04-17 03:55:28,261 DEBUG [Thread-2300] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:28,290 DEBUG [Thread-2296] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:60620 2023-04-17 03:55:28,290 DEBUG [Thread-2296] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 166 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 184 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 189 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1680 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5035 rows 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1656 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 4966 rows 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1664 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 4992 rows 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1667 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5000 rows 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1668 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5001 rows 2023-04-17 03:55:28,291 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:55:28,291 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:60620 2023-04-17 03:55:28,292 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:28,293 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:55:28,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:55:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:28,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:28,298 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703728298"}]},"ts":"1681703728298"} 2023-04-17 03:55:28,299 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:55:28,301 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:55:28,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, UNASSIGN}] 2023-04-17 03:55:28,304 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, UNASSIGN 2023-04-17 03:55:28,304 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:28,304 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703728304"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703728304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703728304"}]},"ts":"1681703728304"} 2023-04-17 03:55:28,305 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:28,357 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=807, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/9a3499fc2c594030a2dae22fbd0e996a 2023-04-17 03:55:28,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:28,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:28,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ed08334fa02b7639086fbab0bf0f1d10, disabling compactions & flushes 2023-04-17 03:55:28,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:28,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9994efaceec44da089af0ae3bad3bc93 2023-04-17 03:55:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:29,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=807 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/6135a954992a449eb7d7dc519c51d724 2023-04-17 03:55:29,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/9a3499fc2c594030a2dae22fbd0e996a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/9a3499fc2c594030a2dae22fbd0e996a 2023-04-17 03:55:29,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/9a3499fc2c594030a2dae22fbd0e996a, entries=150, sequenceid=807, filesize=25.4 K 2023-04-17 03:55:29,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/9994efaceec44da089af0ae3bad3bc93 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9994efaceec44da089af0ae3bad3bc93 2023-04-17 03:55:29,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9994efaceec44da089af0ae3bad3bc93, entries=150, sequenceid=807, filesize=11.9 K 2023-04-17 03:55:29,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/6135a954992a449eb7d7dc519c51d724 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6135a954992a449eb7d7dc519c51d724 2023-04-17 03:55:29,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6135a954992a449eb7d7dc519c51d724, entries=150, sequenceid=807, filesize=11.9 K 2023-04-17 03:55:29,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=20.13 KB/20610 for ed08334fa02b7639086fbab0bf0f1d10 in 1254ms, sequenceid=807, compaction requested=true 2023-04-17 03:55:29,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:29,192 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:29,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:29,192 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. because compaction request was cancelled 2023-04-17 03:55:29,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:29,192 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. because compaction request was cancelled 2023-04-17 03:55:29,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:29,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. after waiting 0 ms 2023-04-17 03:55:29,192 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. because compaction request was cancelled 2023-04-17 03:55:29,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:29,192 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:29,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ed08334fa02b7639086fbab0bf0f1d10 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=A 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=B 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ed08334fa02b7639086fbab0bf0f1d10, store=C 2023-04-17 03:55:29,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:29,602 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172c31c49c8d4b4f7292e7ea057621b00f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172c31c49c8d4b4f7292e7ea057621b00f 2023-04-17 03:55:30,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=813, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c8c2d75dc1f14abfb34628ca9cbb22fc 2023-04-17 03:55:30,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:30,417 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/35438de72c3c4163845a84b3e01e7ef8 2023-04-17 03:55:30,825 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=813 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a6743f5816b9473689d1e1e61b58325d 2023-04-17 03:55:30,829 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/A/c8c2d75dc1f14abfb34628ca9cbb22fc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c8c2d75dc1f14abfb34628ca9cbb22fc 2023-04-17 03:55:30,832 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c8c2d75dc1f14abfb34628ca9cbb22fc, entries=150, sequenceid=813, filesize=25.4 K 2023-04-17 03:55:30,833 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/B/35438de72c3c4163845a84b3e01e7ef8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/35438de72c3c4163845a84b3e01e7ef8 2023-04-17 03:55:30,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/35438de72c3c4163845a84b3e01e7ef8, entries=150, sequenceid=813, filesize=11.9 K 2023-04-17 03:55:30,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/.tmp/C/a6743f5816b9473689d1e1e61b58325d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a6743f5816b9473689d1e1e61b58325d 2023-04-17 03:55:30,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a6743f5816b9473689d1e1e61b58325d, entries=150, sequenceid=813, filesize=11.9 K 2023-04-17 03:55:30,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for ed08334fa02b7639086fbab0bf0f1d10 in 1650ms, sequenceid=813, compaction requested=true 2023-04-17 03:55:30,865 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/700bdead73cb46d7bcc5a6ddec492ac4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/833c5b9b1dff44efa3010b23ea90e05a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5] to archive 2023-04-17 03:55:30,866 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:30,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e3ca79debf8c4056b4a7e4ebc83f4bee 2023-04-17 03:55:30,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/0b7589ba21974eb5bc9c99a4615a6369 2023-04-17 03:55:30,870 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5ce68b498d144ad4bec8334eeaae9c3a 2023-04-17 03:55:30,871 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/79051e3f49224de9b752771cf6225207 2023-04-17 03:55:30,872 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/16634477cec944bc9fb2161466adb5ef 2023-04-17 03:55:30,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/3c8f8e10a7f54447bb34b8715d58597d 2023-04-17 03:55:30,875 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/700bdead73cb46d7bcc5a6ddec492ac4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/700bdead73cb46d7bcc5a6ddec492ac4 2023-04-17 03:55:30,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/2b9de47c589c457081247a69b2d404a4 2023-04-17 03:55:30,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/a7981aa0e64c4ca4a843f5b41138aa4e 2023-04-17 03:55:30,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/8655490edcf24e52a8d3514846ed3e5d 2023-04-17 03:55:30,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/11b9fbb3008a44b1a39dcd9106ccfaeb 2023-04-17 03:55:30,880 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/5c742fed34d04f2386a2f4bd18e7995b 2023-04-17 03:55:30,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/618526125a9e472e8243f59a6189c055 2023-04-17 03:55:30,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/833c5b9b1dff44efa3010b23ea90e05a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/833c5b9b1dff44efa3010b23ea90e05a 2023-04-17 03:55:30,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c0b8e9fcbca64074a6fc937989150cda 2023-04-17 03:55:30,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/dc1cdf1597544889b8ce36d10fcc6835 2023-04-17 03:55:30,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/f2847cfa19f14e6aaaa3a27490d5bc1f 2023-04-17 03:55:30,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/548a602cb4e54ca29ef6e58b5e4d4ced 2023-04-17 03:55:30,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/70edae6c57f74ad8bf076a2b6e3192e5 2023-04-17 03:55:30,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/05fe0bf663694a9e9f08da34bd9d0630, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/c712dc8e80cf44bca83d663610050b18, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d] to archive 2023-04-17 03:55:30,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:30,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d926f6dc2eda4f7a837fce81c83703d5 2023-04-17 03:55:30,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9400336a6e7c49a1ae51d1dc893d5289 2023-04-17 03:55:30,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/adbfb1ad76044a929b0487af7e787ad2 2023-04-17 03:55:30,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dc298fcec754156b27f8a30ff27a2ae 2023-04-17 03:55:30,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/463d00d649664ed0ad2a2d52a9d46533 2023-04-17 03:55:30,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/05fe0bf663694a9e9f08da34bd9d0630 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/05fe0bf663694a9e9f08da34bd9d0630 2023-04-17 03:55:30,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/e3bfc9373e8a4ddfb63d7814e14d035f 2023-04-17 03:55:30,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/dc85aae83cb1441abbaa628c4747214e 2023-04-17 03:55:30,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/5dafea6566e04b19a0acae090eb61cfc 2023-04-17 03:55:30,908 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/cb7d263a6247431b8df8aa439bba9dba 2023-04-17 03:55:30,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f612578965c34f94aa784c49f6c04408 2023-04-17 03:55:30,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/f74e7cdefa4144a989427b8b99024d63 2023-04-17 03:55:30,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9dce5f50caa1415ea9ada8193dbfe612 2023-04-17 03:55:30,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/c712dc8e80cf44bca83d663610050b18 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/c712dc8e80cf44bca83d663610050b18 2023-04-17 03:55:30,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/56ad2564254d4d1eb841f13e44716c91 2023-04-17 03:55:30,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/4b2e0b170b484e559ff1943fc487ec12 2023-04-17 03:55:30,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/20bf1ebdb0164e76808adce9da05f0bf 2023-04-17 03:55:30,918 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/d015377bfcba4851b4cb9ad0861ac573 2023-04-17 03:55:30,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/374db406d6de49ac8b5d9fa95d70443d 2023-04-17 03:55:30,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/acc7faac931b44dc929378733c9f4564, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d0a6f9d5bdb2468d90a73842288c6d9c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1] to archive 2023-04-17 03:55:30,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:30,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/bff919aaa830489a872bb89346fb4ddc 2023-04-17 03:55:30,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1e2c4d65904741a09ee93f2b37a1cce9 2023-04-17 03:55:30,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a3aa5f6827054fcb9f17d0fd2fec3bba 2023-04-17 03:55:30,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cfaf1126ad294bf7a938d166df34e33b 2023-04-17 03:55:30,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/c013206c74c64d7b8cfe5dc18e2c855f 2023-04-17 03:55:30,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/acc7faac931b44dc929378733c9f4564 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/acc7faac931b44dc929378733c9f4564 2023-04-17 03:55:30,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d72b3a683a04467791f7d6d08d27389a 2023-04-17 03:55:30,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/0b82605f98104bc7b06fce413ed27a86 2023-04-17 03:55:30,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/90c1e0ffe65443f3bdfac1ddfc7be519 2023-04-17 03:55:30,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/378e28f333ea4821921b72520febd28f 2023-04-17 03:55:30,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/e76bc9f27c3d4f2cb3b6b792f12387fa 2023-04-17 03:55:30,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/cbff9a7b446f428cb395a40f497ff20e 2023-04-17 03:55:30,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/1dc4dbad5d8943ddabe0c692cfeda107 2023-04-17 03:55:30,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d0a6f9d5bdb2468d90a73842288c6d9c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/d0a6f9d5bdb2468d90a73842288c6d9c 2023-04-17 03:55:30,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/35119223318d40aa93fc3934ad515d2a 2023-04-17 03:55:30,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/ae8dac417f7f4cbd9569021607580c81 2023-04-17 03:55:30,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/b0fc441d0bf7486884312436e1bef962 2023-04-17 03:55:30,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/303e1fadd82043d2bb8dbe0ebf7921f0 2023-04-17 03:55:30,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/7a07ad9e5e674185bb13624ab10dfcf1 2023-04-17 03:55:30,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits/816.seqid, newMaxSeqId=816, maxSeqId=4 2023-04-17 03:55:30,943 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10. 2023-04-17 03:55:30,943 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ed08334fa02b7639086fbab0bf0f1d10: 2023-04-17 03:55:30,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:30,945 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ed08334fa02b7639086fbab0bf0f1d10, regionState=CLOSED 2023-04-17 03:55:30,945 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703730945"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703730945"}]},"ts":"1681703730945"} 2023-04-17 03:55:30,947 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-17 03:55:30,947 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure ed08334fa02b7639086fbab0bf0f1d10, server=jenkins-hbase4.apache.org,36381,1681703614253 in 2.6410 sec 2023-04-17 03:55:30,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-17 03:55:30,948 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ed08334fa02b7639086fbab0bf0f1d10, UNASSIGN in 2.6450 sec 2023-04-17 03:55:30,948 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703730948"}]},"ts":"1681703730948"} 2023-04-17 03:55:30,949 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:55:30,950 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:55:30,951 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6570 sec 2023-04-17 03:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-17 03:55:32,402 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-17 03:55:32,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:55:32,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,405 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 03:55:32,405 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,409 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:32,411 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits] 2023-04-17 03:55:32,413 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/077865f8e38a43bbb9953f16b53cb797 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/077865f8e38a43bbb9953f16b53cb797 2023-04-17 03:55:32,415 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/6147654da0754a719ea7ae15a0407d15 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/6147654da0754a719ea7ae15a0407d15 2023-04-17 03:55:32,416 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/9a3499fc2c594030a2dae22fbd0e996a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/9a3499fc2c594030a2dae22fbd0e996a 2023-04-17 03:55:32,417 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c8c2d75dc1f14abfb34628ca9cbb22fc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c8c2d75dc1f14abfb34628ca9cbb22fc 2023-04-17 03:55:32,418 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c9f7756c00e14d24b2d89fe0e1a161b3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/c9f7756c00e14d24b2d89fe0e1a161b3 2023-04-17 03:55:32,419 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e78ad240801a49fcb36b8412e4a0b6c6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/A/e78ad240801a49fcb36b8412e4a0b6c6 2023-04-17 03:55:32,427 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/35438de72c3c4163845a84b3e01e7ef8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/35438de72c3c4163845a84b3e01e7ef8 2023-04-17 03:55:32,429 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/52787a167d594a6599e009b9020ab3c3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/52787a167d594a6599e009b9020ab3c3 2023-04-17 03:55:32,430 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7327801c0d484235adfd0e151822a4e7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7327801c0d484235adfd0e151822a4e7 2023-04-17 03:55:32,431 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/763791935aa544faba8b205c67bc295d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/763791935aa544faba8b205c67bc295d 2023-04-17 03:55:32,433 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7d8d636607cf4659918673519cefc2ba to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/7d8d636607cf4659918673519cefc2ba 2023-04-17 03:55:32,434 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9994efaceec44da089af0ae3bad3bc93 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/B/9994efaceec44da089af0ae3bad3bc93 2023-04-17 03:55:32,436 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/594b4b696cfb4e838c5d22baa8b43730 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/594b4b696cfb4e838c5d22baa8b43730 2023-04-17 03:55:32,437 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6135a954992a449eb7d7dc519c51d724 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6135a954992a449eb7d7dc519c51d724 2023-04-17 03:55:32,438 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6b745ba4315a42a9ab04dd549ede5eff to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/6b745ba4315a42a9ab04dd549ede5eff 2023-04-17 03:55:32,439 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/75b33601bfee41179bc65e7179803cb6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/75b33601bfee41179bc65e7179803cb6 2023-04-17 03:55:32,440 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a155d48e65234cba88818cab354e72f7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a155d48e65234cba88818cab354e72f7 2023-04-17 03:55:32,440 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a6743f5816b9473689d1e1e61b58325d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/C/a6743f5816b9473689d1e1e61b58325d 2023-04-17 03:55:32,443 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits/816.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10/recovered.edits/816.seqid 2023-04-17 03:55:32,443 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/ed08334fa02b7639086fbab0bf0f1d10 2023-04-17 03:55:32,443 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:32,443 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:55:32,444 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 03:55:32,446 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417019bed7c1db04faf97395d9c25c1ee16 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417019bed7c1db04faf97395d9c25c1ee16 2023-04-17 03:55:32,447 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170a8e60b479d74e60b9aa152c81fa62a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170a8e60b479d74e60b9aa152c81fa62a6 2023-04-17 03:55:32,448 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171bcd7bb07cc3489e9e2971455ff3b305 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171bcd7bb07cc3489e9e2971455ff3b305 2023-04-17 03:55:32,449 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f5b8fa06cb5435c82b72645aa85412f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171f5b8fa06cb5435c82b72645aa85412f 2023-04-17 03:55:32,449 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172a1a59ffd2644833b37619e03ab1ce76 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172a1a59ffd2644833b37619e03ab1ce76 2023-04-17 03:55:32,450 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172c31c49c8d4b4f7292e7ea057621b00f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172c31c49c8d4b4f7292e7ea057621b00f 2023-04-17 03:55:32,451 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173c7d8b35273f4590a29ed93dc896dba4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173c7d8b35273f4590a29ed93dc896dba4 2023-04-17 03:55:32,452 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175cf272ace2f0457a947ce7cd9f7d6bfd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175cf272ace2f0457a947ce7cd9f7d6bfd 2023-04-17 03:55:32,453 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765381b26b2fc44f6a9c09aeabd49c97e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041765381b26b2fc44f6a9c09aeabd49c97e 2023-04-17 03:55:32,454 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041784f50968f1e34f73af9814ca7aea6a33 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041784f50968f1e34f73af9814ca7aea6a33 2023-04-17 03:55:32,455 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041793014d92ee4146689a0dd6f916caf522 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041793014d92ee4146689a0dd6f916caf522 2023-04-17 03:55:32,456 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417afd55c6514044899a719dfde730791c8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417afd55c6514044899a719dfde730791c8 2023-04-17 03:55:32,457 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b4e8e1c372b246e297c5443cd9cbb831 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b4e8e1c372b246e297c5443cd9cbb831 2023-04-17 03:55:32,457 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b5b2f64bce244e598d6ffe3666b035bb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417b5b2f64bce244e598d6ffe3666b035bb 2023-04-17 03:55:32,458 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cd3cf5dd476b4f34a2637e5f0367893e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417cd3cf5dd476b4f34a2637e5f0367893e 2023-04-17 03:55:32,459 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d829f4fdc49b439ab222d923a5981ea9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d829f4fdc49b439ab222d923a5981ea9 2023-04-17 03:55:32,460 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d906ea8402a345c9a1319201490dbd9b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417d906ea8402a345c9a1319201490dbd9b 2023-04-17 03:55:32,461 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3b2a27055f64e4698deee4283e6f9a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3b2a27055f64e4698deee4283e6f9a6 2023-04-17 03:55:32,462 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3cb2921aa0c428997a5c2f90dd45cc7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e3cb2921aa0c428997a5c2f90dd45cc7 2023-04-17 03:55:32,462 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eebfc4534a7d45e6836357fdf15c7393 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eebfc4534a7d45e6836357fdf15c7393 2023-04-17 03:55:32,464 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eef7044a415f4cebaada8b8105d85956 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417eef7044a415f4cebaada8b8105d85956 2023-04-17 03:55:32,465 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f3a502a2a8bd456d93550a93a366a678 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417f3a502a2a8bd456d93550a93a366a678 2023-04-17 03:55:32,465 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:55:32,467 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,468 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:55:32,470 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:55:32,470 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,471 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:55:32,471 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703732471"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:32,472 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:55:32,472 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ed08334fa02b7639086fbab0bf0f1d10, NAME => 'TestAcidGuarantees,,1681703703851.ed08334fa02b7639086fbab0bf0f1d10.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:55:32,472 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:55:32,472 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703732472"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:32,473 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:55:32,474 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,475 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-17 03:55:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-17 03:55:32,506 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-17 03:55:32,517 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=553 (was 540) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=354 (was 350) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=573 (was 656) 2023-04-17 03:55:32,528 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=169, AvailableMemoryMB=573 2023-04-17 03:55:32,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:55:32,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:55:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:32,532 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:55:32,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-17 03:55:32,533 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:55:32,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:32,534 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:32,534 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 empty. 2023-04-17 03:55:32,535 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:32,535 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:32,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:32,941 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:55:32,950 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 82824514d3f4f39e9f4f0e8595870ef8, NAME => 'TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:55:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 82824514d3f4f39e9f4f0e8595870ef8, disabling compactions & flushes 2023-04-17 03:55:33,362 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. after waiting 0 ms 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,362 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:33,364 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:55:33,365 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703733365"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703733365"}]},"ts":"1681703733365"} 2023-04-17 03:55:33,366 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:55:33,366 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:55:33,367 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703733366"}]},"ts":"1681703733366"} 2023-04-17 03:55:33,367 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:55:33,370 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, ASSIGN}] 2023-04-17 03:55:33,371 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, ASSIGN 2023-04-17 03:55:33,371 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:55:33,522 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=82824514d3f4f39e9f4f0e8595870ef8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:33,523 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703733522"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703733522"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703733522"}]},"ts":"1681703733522"} 2023-04-17 03:55:33,524 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:33,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:33,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 82824514d3f4f39e9f4f0e8595870ef8, NAME => 'TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:55:33,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:33,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,679 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,680 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A 2023-04-17 03:55:33,680 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A 2023-04-17 03:55:33,680 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:33,681 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82824514d3f4f39e9f4f0e8595870ef8 columnFamilyName A 2023-04-17 03:55:33,681 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(310): Store=82824514d3f4f39e9f4f0e8595870ef8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:33,681 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,682 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B 2023-04-17 03:55:33,682 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B 2023-04-17 03:55:33,682 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:33,683 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82824514d3f4f39e9f4f0e8595870ef8 columnFamilyName B 2023-04-17 03:55:33,683 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(310): Store=82824514d3f4f39e9f4f0e8595870ef8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:33,683 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,684 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C 2023-04-17 03:55:33,684 DEBUG [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C 2023-04-17 03:55:33,684 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:33,684 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 82824514d3f4f39e9f4f0e8595870ef8 columnFamilyName C 2023-04-17 03:55:33,685 INFO [StoreOpener-82824514d3f4f39e9f4f0e8595870ef8-1] regionserver.HStore(310): Store=82824514d3f4f39e9f4f0e8595870ef8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:33,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:55:33,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:33,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:55:33,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 82824514d3f4f39e9f4f0e8595870ef8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60913301, jitterRate=-0.09232108294963837}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:55:33,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:33,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., pid=49, masterSystemTime=1681703733675 2023-04-17 03:55:33,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:33,692 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=82824514d3f4f39e9f4f0e8595870ef8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:33,692 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703733692"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703733692"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703733692"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703733692"}]},"ts":"1681703733692"} 2023-04-17 03:55:33,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-17 03:55:33,694 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 in 169 msec 2023-04-17 03:55:33,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-17 03:55:33,695 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, ASSIGN in 324 msec 2023-04-17 03:55:33,695 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:55:33,695 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703733695"}]},"ts":"1681703733695"} 2023-04-17 03:55:33,696 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:55:33,699 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:55:33,700 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1690 sec 2023-04-17 03:55:34,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-17 03:55:34,638 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-17 03:55:34,638 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,643 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,644 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,646 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42026, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,647 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:55:34,649 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:55:34,650 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,655 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,655 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,659 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,659 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,663 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,663 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,668 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,668 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,672 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,672 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,676 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,676 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,682 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,683 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,686 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,686 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,693 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,694 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:34,697 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:34,711 DEBUG [hconnection-0x71bba646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,712 DEBUG [hconnection-0x1939f8d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,712 DEBUG [hconnection-0x59336701-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,716 DEBUG [hconnection-0x3abfb243-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,716 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:34,720 DEBUG [hconnection-0x2240f292-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,720 DEBUG [hconnection-0x208161a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,721 DEBUG [hconnection-0x4bd1e522-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,721 DEBUG [hconnection-0x11cabc9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,722 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,722 DEBUG [hconnection-0x795364ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,725 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,725 DEBUG [hconnection-0x3d3e9cad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42028, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,730 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:34,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:34,740 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:34,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:34,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:34,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:34,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:34,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:34,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:34,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:34,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:34,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:34,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:34,752 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:34,752 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:34,752 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:34,752 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:34,771 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 03:55:34,772 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:34,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:34,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:34,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:34,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:34,773 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:34,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:34,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:34,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681703734781, max timestamp=1681703734791Num uniques 859; 2023-04-17 03:55:34,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-17 03:55:34,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681703734781, max timestamp=1681703734791Num uniques 859; 2023-04-17 03:55:34,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-17 03:55:34,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18724, totalHeapSize=42776, min timestamp=1681703734781, max timestamp=1681703734791Num uniques 859; 2023-04-17 03:55:34,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=18724, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-17 03:55:34,793 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0a82896c95724310be5b70a68c378796 2023-04-17 03:55:34,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703794802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703794803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703794804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703794804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703794805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,832 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/8a6756dc905a4a10a4cf84d70ab0cd57 2023-04-17 03:55:34,856 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d16d9bbaf38b46cfb1e021b89937cef6 2023-04-17 03:55:34,863 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0a82896c95724310be5b70a68c378796 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796 2023-04-17 03:55:34,867 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796, entries=200, sequenceid=9, filesize=13.9 K 2023-04-17 03:55:34,867 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/8a6756dc905a4a10a4cf84d70ab0cd57 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57 2023-04-17 03:55:34,870 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57, entries=150, sequenceid=9, filesize=11.6 K 2023-04-17 03:55:34,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d16d9bbaf38b46cfb1e021b89937cef6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6 2023-04-17 03:55:34,873 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6, entries=150, sequenceid=9, filesize=11.6 K 2023-04-17 03:55:34,874 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 82824514d3f4f39e9f4f0e8595870ef8 in 122ms, sequenceid=9, compaction requested=false 2023-04-17 03:55:34,874 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:34,874 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:34,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:34,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:34,874 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,876 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:34,876 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:34,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:34,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26cb8a9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:34,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:34,879 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,880 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:34,881 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:34,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:34,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:34,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:34,884 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:34,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:34,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:34,885 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:34,885 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:34,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:34,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:34,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:34,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:34,891 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:34,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:34,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:34,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,891 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:34,891 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:34,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:34,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:34,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:34,892 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:34,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:34,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=161.02 KB heapSize=441.23 KB 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:34,909 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:34,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703794909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/e60ece98c9df49e694a6fc070df1c098 2023-04-17 03:55:34,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703794910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703794912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703794913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:34,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703794913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:34,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/89416847c27d420891050ed0a4b8ce67 2023-04-17 03:55:34,992 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:34,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:34,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:35,000 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:35,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:35,000 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:35,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,002 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:35,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:35,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:35,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:35,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:35,012 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,012 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:35,012 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:35,012 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:35,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:35,014 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:35,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:35,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@696baf60[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:35,016 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703795015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:35,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,020 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:35,020 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:35,020 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:35,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703795019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703795019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,022 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703795022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703795022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,027 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:35,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:35,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:35,027 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,028 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,028 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,028 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,028 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,028 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:35,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,028 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:35,128 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:35,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:35,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:35,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:35,131 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:35,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:35,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:35,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:35,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:35,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:35,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,138 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@479b25a0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:35,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:35,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,142 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:35,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:35,146 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:35,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:35,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:35,146 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:35,146 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:35,148 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,148 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,148 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,149 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:35,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:35,152 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:35,153 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:35,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:35,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703795218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703795223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703795224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703795225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703795225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,253 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:35,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:35,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:35,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:35,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:35,256 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:35,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:35,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:35,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:35,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:35,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:35,261 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,261 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:35,261 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:35,262 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:35,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dffd7c9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:35,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:35,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:35,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:35,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:35,275 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:35,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:35,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:35,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:35,275 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:35,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:35,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:35,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:35,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:35,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:35,282 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:35,282 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:35,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:35,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:35,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/95c2e45ff1764faa8e328af2c9851dbc 2023-04-17 03:55:35,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/e60ece98c9df49e694a6fc070df1c098 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098 2023-04-17 03:55:35,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098, entries=200, sequenceid=45, filesize=13.9 K 2023-04-17 03:55:35,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/89416847c27d420891050ed0a4b8ce67 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67 2023-04-17 03:55:35,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67, entries=150, sequenceid=45, filesize=11.6 K 2023-04-17 03:55:35,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/95c2e45ff1764faa8e328af2c9851dbc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc 2023-04-17 03:55:35,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc, entries=150, sequenceid=45, filesize=11.6 K 2023-04-17 03:55:35,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~441.19 KB/451776, currentSize=26.84 KB/27480 for 82824514d3f4f39e9f4f0e8595870ef8 in 465ms, sequenceid=45, compaction requested=false 2023-04-17 03:55:35,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:35,382 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:35,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:35,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:35,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:35,388 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:35,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:35,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:35,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,390 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:35,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:35,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:35,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:35,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:35,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:35,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:35,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:35,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:35,397 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:35,397 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:35,398 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:35,398 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:35,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:35,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703735537, max timestamp=1681703735549Num uniques 859; 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703735537, max timestamp=1681703735549Num uniques 859; 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703735537, max timestamp=1681703735549Num uniques 859; 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:35,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:35,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703735563, max timestamp=1681703735594Num uniques 1168; 2023-04-17 03:55:35,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-17 03:55:35,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703735563, max timestamp=1681703735594Num uniques 1168; 2023-04-17 03:55:35,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-17 03:55:35,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703735563, max timestamp=1681703735594Num uniques 1168; 2023-04-17 03:55:35,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=12268, suffix heap size=149440,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=268 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681703735607, max timestamp=1681703735633Num uniques 1127; 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681703735607, max timestamp=1681703735633Num uniques 1127; 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681703735607, max timestamp=1681703735633Num uniques 1127; 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-17 03:55:35,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-17 03:55:35,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703795661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703795663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703735660, max timestamp=1681703735668Num uniques 1236; 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=10806, suffix heap size=156512,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=236 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703735660, max timestamp=1681703735668Num uniques 1236; 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=10806, suffix heap size=156512,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=236 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681703735660, max timestamp=1681703735668Num uniques 1236; 2023-04-17 03:55:35,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=10806, suffix heap size=156512,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=236 2023-04-17 03:55:35,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703735712, max timestamp=1681703735737Num uniques 1095; 2023-04-17 03:55:35,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703735712, max timestamp=1681703735737Num uniques 1095; 2023-04-17 03:55:35,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:35,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:35,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681703735712, max timestamp=1681703735737Num uniques 1095; 2023-04-17 03:55:35,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-17 03:55:35,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703795804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703795805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703795805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703795806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703795807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,818 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6fe63d30edf2441e81f29ed2059ea971 2023-04-17 03:55:35,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703795914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703795915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703795915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703795916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:35,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:35,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703795916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,091 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-17 03:55:36,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703796120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703796120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703796122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703796122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703796123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,230 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/9691e893b95a48818de67f53902ea7d1 2023-04-17 03:55:36,242 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/8e4e126ab01d4af9b2c7148c96466f65 2023-04-17 03:55:36,247 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6fe63d30edf2441e81f29ed2059ea971 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971 2023-04-17 03:55:36,250 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971, entries=100, sequenceid=52, filesize=9.3 K 2023-04-17 03:55:36,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/9691e893b95a48818de67f53902ea7d1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1 2023-04-17 03:55:36,254 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1, entries=100, sequenceid=52, filesize=9.3 K 2023-04-17 03:55:36,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/8e4e126ab01d4af9b2c7148c96466f65 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65 2023-04-17 03:55:36,260 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65, entries=100, sequenceid=52, filesize=9.3 K 2023-04-17 03:55:36,260 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=154.31 KB/158010 for 82824514d3f4f39e9f4f0e8595870ef8 in 862ms, sequenceid=52, compaction requested=true 2023-04-17 03:55:36,260 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:36,260 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:36,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:36,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:36,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:36,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:36,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,262 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,263 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a319f2f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:36,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:36,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:36,266 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:36,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:36,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:36,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:36,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:36,276 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:36,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:36,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:36,276 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:36,276 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:36,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:36,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:36,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:36,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:36,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,282 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:36,282 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:36,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:36,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:36,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:36,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:36,382 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:36,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:36,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:36,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:36,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:36,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:36,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:36,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:36,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:36,391 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,391 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:36,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:36,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:36,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:36,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:36,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:36,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:36,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:36,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:36,395 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=154.31 KB heapSize=432 KB 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:36,395 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:36,413 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/198d1f522c854f4e95f94148d79f6e01 2023-04-17 03:55:36,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:36,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:36,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703796434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703796436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703796437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703796441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703796442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,459 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f647007c7bf8421a99b09476318640ee 2023-04-17 03:55:36,483 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:55:36,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,554 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703796545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703796546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703796546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703796552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703796760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703796760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703796760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703796761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,877 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b3b41821c08b43908a37ba55f9f2c33e 2023-04-17 03:55:36,888 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/198d1f522c854f4e95f94148d79f6e01 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01 2023-04-17 03:55:36,892 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01, entries=150, sequenceid=158, filesize=11.8 K 2023-04-17 03:55:36,893 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f647007c7bf8421a99b09476318640ee as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee 2023-04-17 03:55:36,913 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee, entries=150, sequenceid=158, filesize=11.8 K 2023-04-17 03:55:36,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b3b41821c08b43908a37ba55f9f2c33e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e 2023-04-17 03:55:36,940 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e, entries=150, sequenceid=158, filesize=11.8 K 2023-04-17 03:55:36,942 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~431.95 KB/442320, currentSize=26.84 KB/27480 for 82824514d3f4f39e9f4f0e8595870ef8 in 547ms, sequenceid=158, compaction requested=true 2023-04-17 03:55:36,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:36,942 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:36,942 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:36,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,945 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e953da6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:36,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:36,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:36,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:36,949 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,949 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:36,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:36,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:36,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,950 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:36,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:36,953 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:36,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:36,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:36,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:36,953 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:36,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:36,956 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,956 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:36,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:36,956 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:36,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:36,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:36,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:36,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:36,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:36,961 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:36,961 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:36,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:36,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:36,962 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:36,962 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,962 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:36,962 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:36,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:36,987 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:36,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:37,061 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:37,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1703f3467d94434896beda13a29c5745 2023-04-17 03:55:37,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:37,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:37,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:37,080 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:37,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:37,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:37,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,082 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:37,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,104 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:37,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703737098, max timestamp=1681703737115Num uniques 859; 2023-04-17 03:55:37,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:37,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703737098, max timestamp=1681703737115Num uniques 859; 2023-04-17 03:55:37,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:37,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703737098, max timestamp=1681703737115Num uniques 859; 2023-04-17 03:55:37,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:37,122 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:37,122 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:37,135 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:37,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:37,137 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:37,137 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:37,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:37,140 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403da5f1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:37,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:37,142 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:37,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,143 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:37,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:37,149 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:37,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:37,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:37,152 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:37,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:37,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:37,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:37,152 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:37,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:37,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:37,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:37,159 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:37,159 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:37,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703797175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703737169, max timestamp=1681703737192Num uniques 1018; 2023-04-17 03:55:37,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703737169, max timestamp=1681703737192Num uniques 1018; 2023-04-17 03:55:37,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 03:55:37,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681703737169, max timestamp=1681703737192Num uniques 1018; 2023-04-17 03:55:37,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 03:55:37,196 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-17 03:55:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703797176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,259 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:37,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:37,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:37,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:37,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:37,263 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:37,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:37,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:37,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:37,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:37,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:37,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,268 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:37,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:37,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:37,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703797258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,274 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,274 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:37,274 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:37,274 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:37,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:37,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703797262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,278 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:37,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330f73a0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:37,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:37,280 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,281 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:37,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:37,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:37,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:37,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:37,285 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:37,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:37,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:37,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:37,285 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:37,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,287 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703797274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,293 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,294 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:37,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:37,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:37,295 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:37,295 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:37,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,295 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,295 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,295 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703797294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703797298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703797375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703797377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,395 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:37,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:37,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703797389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:37,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:37,402 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:37,402 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,402 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:37,402 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:37,402 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,402 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,403 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:37,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:37,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:37,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:37,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:37,405 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:37,406 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,406 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:37,406 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:37,406 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:37,406 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:37,407 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,408 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,409 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,409 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:37,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a15a09[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:37,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:37,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,413 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,414 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:37,414 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:37,414 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:37,418 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:37,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:37,418 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:37,420 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,420 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,421 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,423 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:37,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:37,428 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:37,428 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:37,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703797508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703797508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,528 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:37,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:37,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:37,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:37,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:37,540 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:37,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:37,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:37,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/72b683b3c5004f4688840fa2cfbcbc95 2023-04-17 03:55:37,542 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:37,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:37,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:37,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:37,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:37,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:37,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:37,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:37,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:37,554 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,554 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:37,554 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:37,554 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:37,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:37,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:37,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:37,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d015d53ed7a041e899021d15841030a6 2023-04-17 03:55:37,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,562 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:37,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:37,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@222e2516[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:37,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:37,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,568 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:37,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:37,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:37,572 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:37,572 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,572 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1703f3467d94434896beda13a29c5745 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:37,572 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:37,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:37,572 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:37,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745, entries=250, sequenceid=168, filesize=16.4 K 2023-04-17 03:55:37,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/72b683b3c5004f4688840fa2cfbcbc95 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95 2023-04-17 03:55:37,581 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,581 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,581 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:37,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:37,592 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:37,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 03:55:37,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:37,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703797589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d015d53ed7a041e899021d15841030a6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6 2023-04-17 03:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:37,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:37,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,598 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,598 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:37,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,598 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:37,599 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:37,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:37,598 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:37,599 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:37,599 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:37,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:37,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:37,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703797592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703797599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6, entries=150, sequenceid=168, filesize=11.8 K 2023-04-17 03:55:37,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 82824514d3f4f39e9f4f0e8595870ef8 in 650ms, sequenceid=168, compaction requested=true 2023-04-17 03:55:37,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:37,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:37,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:37,637 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:37,637 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:37,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-17 03:55:37,639 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66846 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:37,640 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:37,640 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,640 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=65.3 K 2023-04-17 03:55:37,640 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0a82896c95724310be5b70a68c378796, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681703734730 2023-04-17 03:55:37,641 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e60ece98c9df49e694a6fc070df1c098, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681703734798 2023-04-17 03:55:37,641 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:37,641 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:37,641 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,641 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=56.0 K 2023-04-17 03:55:37,641 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6fe63d30edf2441e81f29ed2059ea971, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703734910 2023-04-17 03:55:37,642 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 198d1f522c854f4e95f94148d79f6e01, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681703735767 2023-04-17 03:55:37,642 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8a6756dc905a4a10a4cf84d70ab0cd57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681703734730 2023-04-17 03:55:37,642 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1703f3467d94434896beda13a29c5745, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:37,643 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 89416847c27d420891050ed0a4b8ce67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681703734798 2023-04-17 03:55:37,643 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9691e893b95a48818de67f53902ea7d1, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703734910 2023-04-17 03:55:37,644 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f647007c7bf8421a99b09476318640ee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681703735767 2023-04-17 03:55:37,644 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 72b683b3c5004f4688840fa2cfbcbc95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:37,659 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#384 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:37,666 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#385 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:37,699 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:37,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:37,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:37,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:37,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:37,760 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:37,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:37,760 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:37,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,762 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:37,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:37,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:37,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:37,765 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/2b2b83dd80c140b5ae8a5be6c1e28044 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2b2b83dd80c140b5ae8a5be6c1e28044 2023-04-17 03:55:37,769 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into 2b2b83dd80c140b5ae8a5be6c1e28044(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:37,769 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:37,769 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=11, startTime=1681703737637; duration=0sec 2023-04-17 03:55:37,770 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:37,770 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:37,771 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57386 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:37,771 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:37,771 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,771 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=56.0 K 2023-04-17 03:55:37,772 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d16d9bbaf38b46cfb1e021b89937cef6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681703734730 2023-04-17 03:55:37,772 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 95c2e45ff1764faa8e328af2c9851dbc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681703734798 2023-04-17 03:55:37,772 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8e4e126ab01d4af9b2c7148c96466f65, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703734910 2023-04-17 03:55:37,773 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b3b41821c08b43908a37ba55f9f2c33e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681703735767 2023-04-17 03:55:37,773 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d015d53ed7a041e899021d15841030a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:37,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:37,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:37,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:37,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:37,787 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:37,787 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:37,787 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:37,788 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=134.18 KB heapSize=374.77 KB 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:37,789 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:37,790 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#386 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:37,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:37,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:37,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703797866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,889 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/a60ffcc5e4834747b14e71ae5a2e1058 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/a60ffcc5e4834747b14e71ae5a2e1058 2023-04-17 03:55:37,894 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into a60ffcc5e4834747b14e71ae5a2e1058(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:37,894 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:37,894 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=11, startTime=1681703737637; duration=0sec 2023-04-17 03:55:37,894 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:37,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703797881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703797897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,914 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/f181bf8507504204adafc977564cb169 2023-04-17 03:55:37,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703797926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,938 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703797930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:37,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:37,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703797982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703798000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,128 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/9edd1a973da1461da8881be3786c3ab9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9edd1a973da1461da8881be3786c3ab9 2023-04-17 03:55:38,132 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into 9edd1a973da1461da8881be3786c3ab9(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:38,132 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:38,133 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=11, startTime=1681703737637; duration=0sec 2023-04-17 03:55:38,133 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:38,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703798188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703798214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,381 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/9ac48ca0ef60478a96f41bbccc44f74d 2023-04-17 03:55:38,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703798413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703798437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703798444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703798493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:38,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703798525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,796 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/6a13eda52aea48b8baff98e7f6c00dd1 2023-04-17 03:55:38,804 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/f181bf8507504204adafc977564cb169 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169 2023-04-17 03:55:38,807 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169, entries=150, sequenceid=221, filesize=11.8 K 2023-04-17 03:55:38,808 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/9ac48ca0ef60478a96f41bbccc44f74d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d 2023-04-17 03:55:38,813 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d, entries=150, sequenceid=221, filesize=11.8 K 2023-04-17 03:55:38,813 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/6a13eda52aea48b8baff98e7f6c00dd1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1 2023-04-17 03:55:38,833 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1, entries=150, sequenceid=221, filesize=11.8 K 2023-04-17 03:55:38,834 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~374.72 KB/383712, currentSize=46.96 KB/48090 for 82824514d3f4f39e9f4f0e8595870ef8 in 1046ms, sequenceid=221, compaction requested=false 2023-04-17 03:55:38,834 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:38,834 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:38,834 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,837 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:38,838 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:38,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:38,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:38,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f910e8f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:38,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:38,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,843 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:38,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:38,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:38,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:38,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:38,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:38,859 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:38,860 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:38,860 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:38,860 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:38,860 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:38,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,862 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:38,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:38,862 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:38,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:38,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:38,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:38,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:38,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:38,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:38,867 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:38,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:38,868 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:38,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:38,968 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:38,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:38,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:38,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:38,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:38,975 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:38,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:38,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:38,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:38,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:38,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:38,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:38,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:38,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:38,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:38,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:38,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:38,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:38,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:38,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:38,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:38,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:38,992 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:38,992 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:38,992 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:38,993 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:38,993 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:39,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:39,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,039 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/29a0020de3e74ea39e5ec60d7fc0892c 2023-04-17 03:55:39,059 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/fdb77080873b40999d048f2c79004ad4 2023-04-17 03:55:39,072 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/ea7d0eb85e3f4a36884f74b2453582d8 2023-04-17 03:55:39,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/29a0020de3e74ea39e5ec60d7fc0892c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c 2023-04-17 03:55:39,080 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c, entries=150, sequenceid=233, filesize=11.8 K 2023-04-17 03:55:39,082 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/fdb77080873b40999d048f2c79004ad4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4 2023-04-17 03:55:39,085 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4, entries=150, sequenceid=233, filesize=11.8 K 2023-04-17 03:55:39,092 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/ea7d0eb85e3f4a36884f74b2453582d8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8 2023-04-17 03:55:39,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8, entries=150, sequenceid=233, filesize=11.8 K 2023-04-17 03:55:39,110 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 82824514d3f4f39e9f4f0e8595870ef8 in 118ms, sequenceid=233, compaction requested=true 2023-04-17 03:55:39,111 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:39,111 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,111 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681703739048, max timestamp=1681703739094Num uniques 859; 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681703739048, max timestamp=1681703739094Num uniques 859; 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681703739048, max timestamp=1681703739094Num uniques 859; 2023-04-17 03:55:39,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@474beeae[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,115 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,119 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,122 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,122 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,122 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:39,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=53.67 KB heapSize=141.23 KB 2023-04-17 03:55:39,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:39,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:39,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:39,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,125 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,127 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,133 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,133 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,134 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,134 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703739165, max timestamp=1681703739192Num uniques 859; 2023-04-17 03:55:39,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703739165, max timestamp=1681703739192Num uniques 859; 2023-04-17 03:55:39,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703739165, max timestamp=1681703739192Num uniques 859; 2023-04-17 03:55:39,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:39,234 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,237 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,237 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,237 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,237 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,238 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,238 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,240 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,245 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1865855c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,246 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,247 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,256 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,256 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,257 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,265 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,265 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703799269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703799274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,365 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,369 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,370 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,370 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,371 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,380 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bb015d3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,382 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,388 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,388 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,390 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,391 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703799387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703799389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,394 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,395 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,395 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,395 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,395 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,394 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703799424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703799452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703799454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,495 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,498 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,499 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,499 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,502 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,503 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,505 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,507 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cd29f5c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,512 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,512 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,518 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,518 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,521 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,534 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,534 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,534 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/9555a52f36ef4203b653891e19442234 2023-04-17 03:55:39,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/02c316bbd6af468da56f8a6b062b231f 2023-04-17 03:55:39,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d3f6243d145843b993e9125a804a60c2 2023-04-17 03:55:39,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/9555a52f36ef4203b653891e19442234 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234 2023-04-17 03:55:39,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 03:55:39,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/02c316bbd6af468da56f8a6b062b231f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f 2023-04-17 03:55:39,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 03:55:39,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d3f6243d145843b993e9125a804a60c2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2 2023-04-17 03:55:39,601 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2, entries=150, sequenceid=257, filesize=11.8 K 2023-04-17 03:55:39,602 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~141.19 KB/144576, currentSize=134.18 KB/137400 for 82824514d3f4f39e9f4f0e8595870ef8 in 479ms, sequenceid=257, compaction requested=true 2023-04-17 03:55:39,602 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:39,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:39,602 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:39,602 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:39,602 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:39,603 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:39,604 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48366 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:39,604 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:39,604 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,604 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9edd1a973da1461da8881be3786c3ab9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=47.2 K 2023-04-17 03:55:39,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:39,606 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 03:55:39,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:39,606 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,606 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:39,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,606 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48366 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:39,607 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:39,607 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9edd1a973da1461da8881be3786c3ab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:39,607 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:39,607 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,607 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2b2b83dd80c140b5ae8a5be6c1e28044, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=47.2 K 2023-04-17 03:55:39,607 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:39,607 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f181bf8507504204adafc977564cb169, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703737255 2023-04-17 03:55:39,608 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2b2b83dd80c140b5ae8a5be6c1e28044, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:39,608 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 29a0020de3e74ea39e5ec60d7fc0892c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681703737863 2023-04-17 03:55:39,608 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9ac48ca0ef60478a96f41bbccc44f74d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703737255 2023-04-17 03:55:39,609 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9555a52f36ef4203b653891e19442234, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:39,609 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fdb77080873b40999d048f2c79004ad4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681703737863 2023-04-17 03:55:39,610 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 02c316bbd6af468da56f8a6b062b231f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:39,630 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#397 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:39,632 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:39,634 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,644 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,646 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703799637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703799644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,663 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,663 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,663 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,663 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,665 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f436f6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,670 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/33622698072b4d1498a52fb2e1efb597 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/33622698072b4d1498a52fb2e1efb597 2023-04-17 03:55:39,673 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,673 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,674 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,674 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into 33622698072b4d1498a52fb2e1efb597(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:39,674 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:39,675 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=12, startTime=1681703739602; duration=0sec 2023-04-17 03:55:39,675 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:39,675 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:39,675 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,675 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,675 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,676 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48366 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:39,676 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,676 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,676 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/a60ffcc5e4834747b14e71ae5a2e1058, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=47.2 K 2023-04-17 03:55:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,677 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a60ffcc5e4834747b14e71ae5a2e1058, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681703736436 2023-04-17 03:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,677 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6a13eda52aea48b8baff98e7f6c00dd1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703737255 2023-04-17 03:55:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,677 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ea7d0eb85e3f4a36884f74b2453582d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1681703737863 2023-04-17 03:55:39,678 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d3f6243d145843b993e9125a804a60c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,678 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,678 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,678 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,678 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,678 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,680 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,681 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,686 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,686 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,686 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,686 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,687 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,689 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,690 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,686 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,691 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#399 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:39,709 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/bb60ebf894634c5cb7f5468c5b8ded05 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bb60ebf894634c5cb7f5468c5b8ded05 2023-04-17 03:55:39,713 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into bb60ebf894634c5cb7f5468c5b8ded05(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:39,713 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:39,713 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=12, startTime=1681703739603; duration=0sec 2023-04-17 03:55:39,713 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:39,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703799753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703799753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,790 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,794 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,801 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,808 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,808 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,808 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,808 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,808 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,811 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,812 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,812 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608f1651[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,815 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,815 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,816 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,818 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,820 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,823 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,823 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,823 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,834 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,841 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,842 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,842 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,842 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:39,942 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:39,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:39,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:39,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:39,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:39,947 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:39,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:39,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:39,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,949 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:39,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:39,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:39,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:39,951 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,951 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:39,951 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:39,951 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:39,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:39,952 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:39,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:39,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:39,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bda891e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:39,957 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:39,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:39,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:39,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:39,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:39,966 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:39,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:39,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:39,966 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:39,966 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:39,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:39,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:39,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42092 deadline: 1681703799961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:39,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42062 deadline: 1681703799962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:39,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:39,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:39,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,974 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,974 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,974 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:39,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:39,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:39,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:39,975 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:39,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:39,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:39,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:39,975 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:39,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:39,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/fa919bbaf83648b088ebfa9a24269f86 2023-04-17 03:55:40,075 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,076 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1af8e0b109e34dbda78421d1e6ea15c9 2023-04-17 03:55:40,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,093 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,095 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,096 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,097 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/2ce69741a0004fa582e6f1da68f0304e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2ce69741a0004fa582e6f1da68f0304e 2023-04-17 03:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,102 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into 2ce69741a0004fa582e6f1da68f0304e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:40,102 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:40,102 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=12, startTime=1681703739602; duration=0sec 2023-04-17 03:55:40,102 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:40,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,150 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,150 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c5ca6908f88e4bebb50c4fbd58dfc5b0 2023-04-17 03:55:40,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,157 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732adb46[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/fa919bbaf83648b088ebfa9a24269f86 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86 2023-04-17 03:55:40,169 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,169 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,169 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,169 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,169 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,169 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,171 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86, entries=200, sequenceid=294, filesize=14.3 K 2023-04-17 03:55:40,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1af8e0b109e34dbda78421d1e6ea15c9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9 2023-04-17 03:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,184 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,184 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,186 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,186 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,186 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,187 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,188 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:40,193 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,194 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,194 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9, entries=150, sequenceid=294, filesize=11.9 K 2023-04-17 03:55:40,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c5ca6908f88e4bebb50c4fbd58dfc5b0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0 2023-04-17 03:55:40,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0, entries=150, sequenceid=294, filesize=11.9 K 2023-04-17 03:55:40,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 82824514d3f4f39e9f4f0e8595870ef8 in 603ms, sequenceid=294, compaction requested=false 2023-04-17 03:55:40,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:40,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:40,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:40,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:40,294 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,298 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,304 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,304 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,304 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,304 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,316 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,316 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,316 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703740312, max timestamp=1681703740326Num uniques 859; 2023-04-17 03:55:40,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:40,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703740312, max timestamp=1681703740326Num uniques 859; 2023-04-17 03:55:40,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:40,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703740312, max timestamp=1681703740326Num uniques 859; 2023-04-17 03:55:40,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c0053[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,332 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,335 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,335 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,337 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,342 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,347 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,347 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,347 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703800384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703800388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,447 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,450 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,452 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,454 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,455 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,456 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb9a34f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,465 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,465 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,475 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,475 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,480 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,482 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,482 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,482 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703800489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703800493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,582 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,586 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,586 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,588 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,589 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,589 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,590 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d44bff[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,594 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,598 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,599 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,601 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,609 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,611 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,611 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,611 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,612 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,612 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/4c236b1df9a2407e945c8cc3d9e802df 2023-04-17 03:55:40,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703800697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,703 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/44db02f1140f4c4dbc1cec9bfbeeed0a 2023-04-17 03:55:40,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:40,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703800699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,713 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,719 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,721 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,723 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,723 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,723 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,723 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,723 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,725 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,725 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44814c68[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,728 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,728 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,731 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,736 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,736 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,736 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,736 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,738 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,738 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,740 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,740 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,740 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,743 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,745 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,745 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,745 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,745 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,845 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,849 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,849 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,850 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,851 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,851 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,853 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,853 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,853 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,853 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,853 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,854 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,856 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,856 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,857 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f94233[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,859 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,859 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,859 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,861 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:40,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,863 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,863 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,863 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,866 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,866 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,869 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,870 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,871 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,871 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:40,971 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:40,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:40,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:40,974 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:40,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:40,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:40,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,976 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,976 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:40,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:40,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:40,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:40,978 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,978 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:40,978 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:40,978 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:40,978 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:40,979 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:40,980 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,980 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,981 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:40,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71892ef8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:40,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:40,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:40,985 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:40,986 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:40,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:40,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:40,987 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:40,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:40,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:40,987 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:40,987 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:40,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:40,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:40,990 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:40,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:40,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:40,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:40,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:40,994 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:40,994 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:40,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:40,994 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:40,994 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,994 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:40,994 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:40,994 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:40,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:40,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:41,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703801004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703801008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,094 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:41,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:41,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:41,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:41,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:41,097 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:41,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:41,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:41,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,098 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,105 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,105 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:41,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:41,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:41,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:41,109 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:41,109 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,110 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:41,110 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:41,110 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:41,110 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:41,112 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,113 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:41,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/7655aceed82540b48d53fa04e2f0651a 2023-04-17 03:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:41,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32411885[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:41,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:41,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,118 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:41,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:41,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,118 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:41,119 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/4c236b1df9a2407e945c8cc3d9e802df as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df 2023-04-17 03:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,120 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:41,120 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df, entries=100, sequenceid=307, filesize=9.5 K 2023-04-17 03:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:41,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/44db02f1140f4c4dbc1cec9bfbeeed0a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a 2023-04-17 03:55:41,124 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:41,124 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:41,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:41,124 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:41,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:41,125 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:41,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a, entries=100, sequenceid=307, filesize=9.5 K 2023-04-17 03:55:41,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/7655aceed82540b48d53fa04e2f0651a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a 2023-04-17 03:55:41,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,132 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,134 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,135 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:41,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:41,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:41,135 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:41,135 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:41,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a, entries=100, sequenceid=307, filesize=9.5 K 2023-04-17 03:55:41,137 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 82824514d3f4f39e9f4f0e8595870ef8 in 863ms, sequenceid=307, compaction requested=true 2023-04-17 03:55:41,137 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:41,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:41,137 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:41,137 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:41,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:41,137 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:41,138 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:41,138 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:41,138 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,138 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/33622698072b4d1498a52fb2e1efb597, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=35.9 K 2023-04-17 03:55:41,138 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34322 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:41,138 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:41,138 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,138 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2ce69741a0004fa582e6f1da68f0304e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=33.5 K 2023-04-17 03:55:41,139 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 33622698072b4d1498a52fb2e1efb597, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:41,139 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2ce69741a0004fa582e6f1da68f0304e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:41,139 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting fa919bbaf83648b088ebfa9a24269f86, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681703739250 2023-04-17 03:55:41,139 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1af8e0b109e34dbda78421d1e6ea15c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681703739250 2023-04-17 03:55:41,140 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 4c236b1df9a2407e945c8cc3d9e802df, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739642 2023-04-17 03:55:41,140 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 44db02f1140f4c4dbc1cec9bfbeeed0a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739642 2023-04-17 03:55:41,145 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:41,153 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:41,169 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/0d27f6dce5164e5e9986c9bb55659f61 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0d27f6dce5164e5e9986c9bb55659f61 2023-04-17 03:55:41,173 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into 0d27f6dce5164e5e9986c9bb55659f61(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:41,173 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:41,173 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=13, startTime=1681703741137; duration=0sec 2023-04-17 03:55:41,173 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:41,173 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:41,174 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34322 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:41,174 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:41,174 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,174 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bb60ebf894634c5cb7f5468c5b8ded05, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=33.5 K 2023-04-17 03:55:41,175 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting bb60ebf894634c5cb7f5468c5b8ded05, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1681703739048 2023-04-17 03:55:41,175 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c5ca6908f88e4bebb50c4fbd58dfc5b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681703739250 2023-04-17 03:55:41,175 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7655aceed82540b48d53fa04e2f0651a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739642 2023-04-17 03:55:41,180 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#407 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:41,189 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/00c96335eb0145e3a0c3c5716d8969ee as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/00c96335eb0145e3a0c3c5716d8969ee 2023-04-17 03:55:41,192 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into 00c96335eb0145e3a0c3c5716d8969ee(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:41,192 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:41,192 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=13, startTime=1681703741137; duration=0sec 2023-04-17 03:55:41,193 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:41,235 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:41,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:41,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:41,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:41,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:41,238 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:41,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:41,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:41,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,239 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,241 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,241 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:41,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:41,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:41,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:41,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:41,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:41,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:41,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:41,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:41,243 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:41,243 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:41,263 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5d554a0d11194235b37134ffad5bc0f2 2023-04-17 03:55:41,279 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/64954c355b4c445e80f8bf7022f3744a 2023-04-17 03:55:41,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:41,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:41,483 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703801479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,485 DEBUG [Thread-2926] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4210 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:41,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703801482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,487 DEBUG [Thread-2924] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4228 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:55:41,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703801483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703801509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703801512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,568 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/8feb391460014957a2095a74a2415510 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/8feb391460014957a2095a74a2415510 2023-04-17 03:55:41,572 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into 8feb391460014957a2095a74a2415510(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:41,573 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:41,573 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=13, startTime=1681703741137; duration=0sec 2023-04-17 03:55:41,573 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:41,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703801592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,694 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d8bf10c104564bac8ea5c8ec642e0da4 2023-04-17 03:55:41,698 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5d554a0d11194235b37134ffad5bc0f2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2 2023-04-17 03:55:41,701 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2, entries=150, sequenceid=345, filesize=11.9 K 2023-04-17 03:55:41,702 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/64954c355b4c445e80f8bf7022f3744a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a 2023-04-17 03:55:41,705 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a, entries=150, sequenceid=345, filesize=11.9 K 2023-04-17 03:55:41,706 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/d8bf10c104564bac8ea5c8ec642e0da4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4 2023-04-17 03:55:41,709 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4, entries=150, sequenceid=345, filesize=11.9 K 2023-04-17 03:55:41,710 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 82824514d3f4f39e9f4f0e8595870ef8 in 467ms, sequenceid=345, compaction requested=false 2023-04-17 03:55:41,710 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:41,710 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,710 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:41,710 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:41,710 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,710 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:41,710 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:41,713 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,713 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,713 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,713 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:41,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:41,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cedf0a9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:41,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:41,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,716 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:41,716 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:41,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:41,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,718 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:41,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:41,721 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:41,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:41,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:41,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:41,721 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:41,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,723 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,723 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,732 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:41,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:41,732 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,732 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,732 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:41,732 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:41,732 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,732 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:41,797 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:41,797 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:41,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/3420f55316d1417d9dd3040cf5280849 2023-04-17 03:55:41,832 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:41,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:41,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:41,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:41,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:41,837 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:41,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:41,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:41,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,839 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,839 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:41,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:41,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:41,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:41,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:41,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:41,841 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:41,842 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,842 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,843 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:41,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146dcddd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:41,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:41,845 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,846 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:41,847 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:41,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,847 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:41,850 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,850 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:41,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:41,852 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:41,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:41,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:41,852 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:41,852 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:41,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,854 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,855 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,856 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:41,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,858 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,858 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:41,858 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:41,858 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703741864, max timestamp=1681703741880Num uniques 859; 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703741864, max timestamp=1681703741880Num uniques 859; 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703741864, max timestamp=1681703741880Num uniques 859; 2023-04-17 03:55:41,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:41,959 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:41,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:41,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:41,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:41,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:41,962 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:41,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:41,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:41,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:41,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:41,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:41,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:41,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:41,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:41,965 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:41,966 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:41,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,969 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703801961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cabdf46[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:41,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:41,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:41,972 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,972 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:41,973 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,973 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:41,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:41,975 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:41,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:41,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:41,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:41,976 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:41,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:41,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,977 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:41,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:41,977 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,979 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:41,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:41,984 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,985 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:41,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:41,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:41,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:41,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:41,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:41,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:41,985 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:41,985 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:41,985 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:41,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703802071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,085 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,089 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,090 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,092 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,092 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,092 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,092 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,093 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,093 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,095 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,096 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16146f66[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,100 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,100 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,100 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,102 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,104 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,104 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,104 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,106 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,107 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,108 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,112 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,112 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,112 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,113 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,112 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,213 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/335d5092ef7b422497c4f6448860b027 2023-04-17 03:55:42,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,222 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,228 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,229 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,229 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,229 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,229 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,230 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,230 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,230 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,231 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,231 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,232 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fbdb5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,235 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,235 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,235 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,235 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,236 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,236 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,240 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,240 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,240 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,243 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,245 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,247 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,247 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,248 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,248 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:42,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/208c69dfc95e43b9ab64675b11adfdbe 2023-04-17 03:55:42,261 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/3420f55316d1417d9dd3040cf5280849 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849 2023-04-17 03:55:42,265 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849, entries=150, sequenceid=357, filesize=11.9 K 2023-04-17 03:55:42,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/335d5092ef7b422497c4f6448860b027 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027 2023-04-17 03:55:42,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027, entries=100, sequenceid=357, filesize=9.5 K 2023-04-17 03:55:42,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/208c69dfc95e43b9ab64675b11adfdbe as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe 2023-04-17 03:55:42,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe, entries=100, sequenceid=357, filesize=9.5 K 2023-04-17 03:55:42,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 82824514d3f4f39e9f4f0e8595870ef8 in 477ms, sequenceid=357, compaction requested=true 2023-04-17 03:55:42,274 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:42,274 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:42,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:42,275 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:42,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:42,275 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:42,276 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:42,276 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:42,276 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:42,276 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:42,276 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,276 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,276 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0d27f6dce5164e5e9986c9bb55659f61, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=33.7 K 2023-04-17 03:55:42,276 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/8feb391460014957a2095a74a2415510, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=36.1 K 2023-04-17 03:55:42,276 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0d27f6dce5164e5e9986c9bb55659f61, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739272 2023-04-17 03:55:42,277 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8feb391460014957a2095a74a2415510, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739272 2023-04-17 03:55:42,277 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 64954c355b4c445e80f8bf7022f3744a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681703740375 2023-04-17 03:55:42,277 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5d554a0d11194235b37134ffad5bc0f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681703740375 2023-04-17 03:55:42,277 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 335d5092ef7b422497c4f6448860b027, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703741476 2023-04-17 03:55:42,278 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 3420f55316d1417d9dd3040cf5280849, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703741476 2023-04-17 03:55:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:42,282 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-17 03:55:42,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:42,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:42,282 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:42,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:42,283 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:42,283 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:42,283 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:42,287 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:42,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1cb700c4d8f04506bff4a3d1875cf88d 2023-04-17 03:55:42,317 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/a18fc6871b19445ca40e43fca604e820 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a18fc6871b19445ca40e43fca604e820 2023-04-17 03:55:42,325 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into a18fc6871b19445ca40e43fca604e820(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:42,326 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:42,326 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=13, startTime=1681703742274; duration=0sec 2023-04-17 03:55:42,326 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:42,326 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:42,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:42,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:42,327 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,327 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/00c96335eb0145e3a0c3c5716d8969ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=33.7 K 2023-04-17 03:55:42,327 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 00c96335eb0145e3a0c3c5716d8969ee, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681703739272 2023-04-17 03:55:42,328 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d8bf10c104564bac8ea5c8ec642e0da4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1681703740375 2023-04-17 03:55:42,328 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 208c69dfc95e43b9ab64675b11adfdbe, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703741476 2023-04-17 03:55:42,341 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:42,348 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42072 deadline: 1681703802351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,356 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,360 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c0ca143830a849218756429dee880ee1 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0ca143830a849218756429dee880ee1 2023-04-17 03:55:42,360 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,360 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,360 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,360 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,360 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417939c9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,362 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,362 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,364 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,364 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,364 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into c0ca143830a849218756429dee880ee1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:42,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,365 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:42,365 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=13, startTime=1681703742275; duration=0sec 2023-04-17 03:55:42,365 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:42,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,369 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,369 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,371 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,371 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,372 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,376 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,377 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,377 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703802452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,477 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,481 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,486 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,487 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,487 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,487 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,487 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,488 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,489 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,491 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f169be0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,500 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,500 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,511 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,511 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,511 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:42,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703802521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703802521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,612 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,615 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,618 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,619 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,619 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,619 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,620 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,620 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,621 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,622 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d352aab[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,623 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,623 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,625 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,626 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,626 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,626 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,626 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,628 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,628 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:42,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,634 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,634 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:42,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703802654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,707 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/c63b4556dbf24f368d016fceb005ec7a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c63b4556dbf24f368d016fceb005ec7a 2023-04-17 03:55:42,710 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into c63b4556dbf24f368d016fceb005ec7a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:42,710 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:42,710 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=13, startTime=1681703742275; duration=0sec 2023-04-17 03:55:42,710 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:42,735 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,738 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,738 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,740 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,740 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,741 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,741 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,742 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:42,742 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:42,742 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,743 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ef1b34[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:42,744 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:42,745 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:42,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:42,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/79fa31bbefbf4b2aa518d691d06169be 2023-04-17 03:55:42,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:42,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,749 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:42,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:42,752 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:42,752 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:42,752 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:42,752 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:42,752 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:42,755 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,755 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,756 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,757 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:42,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:42,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:42,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,760 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,760 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:42,760 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,759 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:42,760 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:42,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:42,760 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:42,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:42,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c96578188d99422ab8b550dbf1cb5d56 2023-04-17 03:55:42,771 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1cb700c4d8f04506bff4a3d1875cf88d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d 2023-04-17 03:55:42,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d, entries=200, sequenceid=393, filesize=14.3 K 2023-04-17 03:55:42,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/79fa31bbefbf4b2aa518d691d06169be as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be 2023-04-17 03:55:42,782 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be, entries=150, sequenceid=393, filesize=11.9 K 2023-04-17 03:55:42,785 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c96578188d99422ab8b550dbf1cb5d56 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56 2023-04-17 03:55:42,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56, entries=150, sequenceid=393, filesize=11.9 K 2023-04-17 03:55:42,788 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 82824514d3f4f39e9f4f0e8595870ef8 in 506ms, sequenceid=393, compaction requested=false 2023-04-17 03:55:42,788 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:42,861 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:42,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:42,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:42,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:42,865 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:42,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:42,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:42,866 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,866 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:42,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:42,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:42,868 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:42,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:42,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:42,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:42,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:42,870 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:42,870 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:42,870 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:42,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:42,871 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:42,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:42,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:42,871 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:42,871 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:42,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:42,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:42,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:42,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:42,872 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:42,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:42,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:43,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703743034, max timestamp=1681703743065Num uniques 859; 2023-04-17 03:55:43,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:43,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703743034, max timestamp=1681703743065Num uniques 859; 2023-04-17 03:55:43,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703743034, max timestamp=1681703743065Num uniques 859; 2023-04-17 03:55:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:43,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:43,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703803141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:43,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703803253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,290 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/7aac72b0522c484fafdef7ed6e8f9baa 2023-04-17 03:55:43,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:43,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703803459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,706 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/89cecdeb56964a35819c0419fa8fed5e 2023-04-17 03:55:43,728 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/3851c3d2e318415bb030146c0ee0ef99 2023-04-17 03:55:43,731 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/7aac72b0522c484fafdef7ed6e8f9baa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa 2023-04-17 03:55:43,735 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa, entries=150, sequenceid=406, filesize=11.9 K 2023-04-17 03:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/89cecdeb56964a35819c0419fa8fed5e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e 2023-04-17 03:55:43,741 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e, entries=150, sequenceid=406, filesize=11.9 K 2023-04-17 03:55:43,742 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/3851c3d2e318415bb030146c0ee0ef99 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99 2023-04-17 03:55:43,745 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99, entries=150, sequenceid=406, filesize=11.9 K 2023-04-17 03:55:43,746 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 82824514d3f4f39e9f4f0e8595870ef8 in 875ms, sequenceid=406, compaction requested=true 2023-04-17 03:55:43,746 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:43,746 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:43,751 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:43,751 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,752 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40159b89[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:43,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:43,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,757 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:43,757 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:43,757 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,757 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:43,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:43,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:43,762 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:43,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:43,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:43,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:43,762 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:43,763 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:43,763 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,764 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:43,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:43,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:43,765 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:43,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:43,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:43,771 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:43,771 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:43,770 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,771 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:43,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:43,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:43,785 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0955d6c4cad14339bbd850f759478761 2023-04-17 03:55:43,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/64cd7bc0b1cb4f6d953818d97bdf5a39 2023-04-17 03:55:43,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:43,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703803823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,871 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:43,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:43,874 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:43,874 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,878 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:43,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,878 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:43,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:43,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:43,879 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:43,879 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:43,879 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:43,880 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:43,880 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:43,882 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:43,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,882 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:43,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b17d1f0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:43,883 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:43,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,885 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:43,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:43,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:43,887 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:43,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:43,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:43,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:43,888 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:43,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:43,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:43,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:43,889 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:43,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,891 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:43,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,897 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:43,897 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:43,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:43,897 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:43,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:43,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:43,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:43,897 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:43,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,898 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:43,898 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:43,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:43,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:43,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:43,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:43,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703803930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:43,998 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:43,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:44,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:44,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:44,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:44,001 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,001 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,003 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:44,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:44,004 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,004 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:44,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:44,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:44,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:44,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:44,005 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:44,005 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:44,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,008 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,009 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:44,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fd3c24b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:44,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:44,010 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,012 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:44,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,012 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:44,014 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:44,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:44,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:44,017 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:44,017 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:44,017 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:44,017 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:44,017 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:44,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,019 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:44,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:44,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:44,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:44,024 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,024 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:44,024 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,024 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,025 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:44,025 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,124 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:44,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:44,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:44,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:44,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:44,128 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:44,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:44,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:44,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,129 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,130 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:44,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:44,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:44,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:44,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:44,132 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,132 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:44,132 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:44,132 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:44,133 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:44,135 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,135 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703804134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,136 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:44,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f255a3e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:44,138 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:44,138 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,139 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:44,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:44,139 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:44,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:44,149 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:44,152 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:44,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:44,153 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:44,153 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:44,153 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:44,154 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,154 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:44,154 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:44,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:44,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:44,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:44,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:44,159 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:44,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:44,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:44,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:44,159 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:44,160 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:44,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b42c0ddecef54bcca81f8c4d80c803aa 2023-04-17 03:55:44,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0955d6c4cad14339bbd850f759478761 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761 2023-04-17 03:55:44,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761, entries=200, sequenceid=443, filesize=14.3 K 2023-04-17 03:55:44,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/64cd7bc0b1cb4f6d953818d97bdf5a39 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39 2023-04-17 03:55:44,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39, entries=150, sequenceid=443, filesize=11.9 K 2023-04-17 03:55:44,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b42c0ddecef54bcca81f8c4d80c803aa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa 2023-04-17 03:55:44,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa, entries=150, sequenceid=443, filesize=11.9 K 2023-04-17 03:55:44,229 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 82824514d3f4f39e9f4f0e8595870ef8 in 455ms, sequenceid=443, compaction requested=true 2023-04-17 03:55:44,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:44,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:44,229 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:44,229 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:44,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:44,229 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:44,232 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:44,232 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:44,232 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,232 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:44,232 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a18fc6871b19445ca40e43fca604e820, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=52.9 K 2023-04-17 03:55:44,232 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:44,233 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,233 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c63b4556dbf24f368d016fceb005ec7a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.1 K 2023-04-17 03:55:44,233 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a18fc6871b19445ca40e43fca604e820, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703740375 2023-04-17 03:55:44,233 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c63b4556dbf24f368d016fceb005ec7a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703740375 2023-04-17 03:55:44,233 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1cb700c4d8f04506bff4a3d1875cf88d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681703741898 2023-04-17 03:55:44,233 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 79fa31bbefbf4b2aa518d691d06169be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681703741898 2023-04-17 03:55:44,234 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 89cecdeb56964a35819c0419fa8fed5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681703742320 2023-04-17 03:55:44,234 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 7aac72b0522c484fafdef7ed6e8f9baa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681703742320 2023-04-17 03:55:44,234 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 64cd7bc0b1cb4f6d953818d97bdf5a39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:44,234 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0955d6c4cad14339bbd850f759478761, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:44,242 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#426 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:44,243 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:44,258 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/d2ddd013aa0d438e9b84cb2b5c9756f6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d2ddd013aa0d438e9b84cb2b5c9756f6 2023-04-17 03:55:44,260 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:44,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:44,262 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into d2ddd013aa0d438e9b84cb2b5c9756f6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:44,262 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:44,262 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=12, startTime=1681703744229; duration=0sec 2023-04-17 03:55:44,262 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:44,262 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:44,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:44,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:44,264 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:44,265 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:44,265 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:44,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,265 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:44,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:44,265 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0ca143830a849218756429dee880ee1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.1 K 2023-04-17 03:55:44,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,265 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c0ca143830a849218756429dee880ee1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681703740375 2023-04-17 03:55:44,266 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c96578188d99422ab8b550dbf1cb5d56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681703741898 2023-04-17 03:55:44,266 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 3851c3d2e318415bb030146c0ee0ef99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1681703742320 2023-04-17 03:55:44,266 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b42c0ddecef54bcca81f8c4d80c803aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:44,266 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:44,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:44,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:44,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:44,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:44,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:44,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:44,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:44,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:44,272 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-17 03:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:44,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:44,273 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:44,273 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:44,273 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:44,297 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:44,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:44,439 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703744520, max timestamp=1681703744544Num uniques 859; 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703744520, max timestamp=1681703744544Num uniques 859; 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703744520, max timestamp=1681703744544Num uniques 859; 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:44,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:44,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703804585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703804586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703804593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,654 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/e866cb3f1f2942f98081b03e9c64b19a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e866cb3f1f2942f98081b03e9c64b19a 2023-04-17 03:55:44,658 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into e866cb3f1f2942f98081b03e9c64b19a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:44,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:44,658 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=12, startTime=1681703744229; duration=0sec 2023-04-17 03:55:44,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:44,693 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/221f9f6dd63848178694d96cc7ba9f8f 2023-04-17 03:55:44,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703804697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703804697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,706 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/372a09ae991c4a4e88fec27fb966599d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/372a09ae991c4a4e88fec27fb966599d 2023-04-17 03:55:44,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703804702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,711 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into 372a09ae991c4a4e88fec27fb966599d(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:44,711 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:44,712 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=12, startTime=1681703744229; duration=0sec 2023-04-17 03:55:44,712 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703804906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703804906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703804910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,104 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/3f8c2ad4e543416ebc369bf5ca0e2318 2023-04-17 03:55:45,116 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=453 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/197590dc87984074a1104b133038bedb 2023-04-17 03:55:45,120 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/221f9f6dd63848178694d96cc7ba9f8f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f 2023-04-17 03:55:45,123 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f, entries=150, sequenceid=453, filesize=11.9 K 2023-04-17 03:55:45,124 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/3f8c2ad4e543416ebc369bf5ca0e2318 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318 2023-04-17 03:55:45,130 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318, entries=150, sequenceid=453, filesize=11.9 K 2023-04-17 03:55:45,130 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/197590dc87984074a1104b133038bedb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb 2023-04-17 03:55:45,137 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb, entries=150, sequenceid=453, filesize=11.9 K 2023-04-17 03:55:45,138 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 82824514d3f4f39e9f4f0e8595870ef8 in 866ms, sequenceid=453, compaction requested=false 2023-04-17 03:55:45,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:45,138 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3804d80[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,151 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,152 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,153 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,156 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,156 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,158 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,164 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,164 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,164 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,164 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,165 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,165 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:45,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-17 03:55:45,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703744562, max timestamp=1681703745221Num uniques 1118; 2023-04-17 03:55:45,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:45,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703744562, max timestamp=1681703745221Num uniques 1118; 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:45,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703744562, max timestamp=1681703745221Num uniques 1118; 2023-04-17 03:55:45,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:45,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:45,224 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703745244, max timestamp=1681703745260Num uniques 859; 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703745244, max timestamp=1681703745260Num uniques 859; 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703745244, max timestamp=1681703745260Num uniques 859; 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:45,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:45,265 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,268 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,270 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,270 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,272 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,274 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad8451[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,279 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,279 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,281 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,281 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,281 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,283 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,286 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,286 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,286 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:45,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703745294, max timestamp=1681703745308Num uniques 1018; 2023-04-17 03:55:45,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:45,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703745294, max timestamp=1681703745308Num uniques 1018; 2023-04-17 03:55:45,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:45,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681703745294, max timestamp=1681703745308Num uniques 1018; 2023-04-17 03:55:45,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-17 03:55:45,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703745344, max timestamp=1681703745353Num uniques 1077; 2023-04-17 03:55:45,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703745344, max timestamp=1681703745353Num uniques 1077; 2023-04-17 03:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681703745344, max timestamp=1681703745353Num uniques 1077; 2023-04-17 03:55:45,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-17 03:55:45,386 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,391 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,391 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,397 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,397 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681703745373, max timestamp=1681703745396Num uniques 1036; 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-17 03:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681703745373, max timestamp=1681703745396Num uniques 1036; 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681703745373, max timestamp=1681703745396Num uniques 1036; 2023-04-17 03:55:45,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-17 03:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75dc955[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,399 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,399 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,400 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,401 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,402 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,402 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,405 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,405 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,405 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,412 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,415 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:45,415 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,414 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,415 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703805443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703805444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703745439, max timestamp=1681703745449Num uniques 1145; 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703745439, max timestamp=1681703745449Num uniques 1145; 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681703745439, max timestamp=1681703745449Num uniques 1145; 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-17 03:55:45,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-17 03:55:45,515 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,518 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,519 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:45,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,522 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,522 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,522 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,522 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681703745510, max timestamp=1681703745523Num uniques 1054; 2023-04-17 03:55:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 03:55:45,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681703745510, max timestamp=1681703745523Num uniques 1054; 2023-04-17 03:55:45,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 03:55:45,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681703745510, max timestamp=1681703745523Num uniques 1054; 2023-04-17 03:55:45,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-17 03:55:45,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,526 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,526 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,526 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42ec872a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,527 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,527 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,529 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,529 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,529 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,530 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,533 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,533 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,533 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,533 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,533 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,535 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,537 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,540 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:45,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,540 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,540 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703805585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703805585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703805586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703805586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703745565, max timestamp=1681703745592Num uniques 1113; 2023-04-17 03:55:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-17 03:55:45,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703745565, max timestamp=1681703745592Num uniques 1113; 2023-04-17 03:55:45,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-17 03:55:45,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681703745565, max timestamp=1681703745592Num uniques 1113; 2023-04-17 03:55:45,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-17 03:55:45,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/4499ff11e25243ef9986004d0918393c 2023-04-17 03:55:45,640 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,645 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/158843ca54aa4bf5b96e0d1353f8824a 2023-04-17 03:55:45,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:45,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,652 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,652 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,652 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,652 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,653 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,653 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,654 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,654 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,655 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43013eba[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,656 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/84312d5ee97f43e597fdf626bf2e0d2f 2023-04-17 03:55:45,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,658 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,658 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,658 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,658 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,660 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,660 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/4499ff11e25243ef9986004d0918393c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c 2023-04-17 03:55:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,663 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,663 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,663 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,664 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,664 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c, entries=200, sequenceid=493, filesize=14.3 K 2023-04-17 03:55:45,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,666 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/158843ca54aa4bf5b96e0d1353f8824a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a 2023-04-17 03:55:45,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,666 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:45,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,670 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,671 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,671 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,671 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a, entries=150, sequenceid=493, filesize=11.9 K 2023-04-17 03:55:45,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/84312d5ee97f43e597fdf626bf2e0d2f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f 2023-04-17 03:55:45,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f, entries=150, sequenceid=493, filesize=11.9 K 2023-04-17 03:55:45,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=114.05 KB/116790 for 82824514d3f4f39e9f4f0e8595870ef8 in 454ms, sequenceid=493, compaction requested=true 2023-04-17 03:55:45,676 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:45,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:45,676 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:45,676 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:45,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:45,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:45,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:45,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=120.76 KB heapSize=344.77 KB 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:45,679 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:45,680 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:45,680 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:45,680 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:45,680 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:45,680 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,680 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,680 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d2ddd013aa0d438e9b84cb2b5c9756f6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=38.7 K 2023-04-17 03:55:45,680 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e866cb3f1f2942f98081b03e9c64b19a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=36.3 K 2023-04-17 03:55:45,681 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d2ddd013aa0d438e9b84cb2b5c9756f6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:45,681 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e866cb3f1f2942f98081b03e9c64b19a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:45,681 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 221f9f6dd63848178694d96cc7ba9f8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703743803 2023-04-17 03:55:45,682 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3f8c2ad4e543416ebc369bf5ca0e2318, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703743803 2023-04-17 03:55:45,682 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 4499ff11e25243ef9986004d0918393c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744562 2023-04-17 03:55:45,682 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 158843ca54aa4bf5b96e0d1353f8824a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744592 2023-04-17 03:55:45,701 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:45,702 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#437 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703805702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703805703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703805704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703805709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703805714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,727 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/d99baeb997b04118ac98daacf8ad7d22 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d99baeb997b04118ac98daacf8ad7d22 2023-04-17 03:55:45,731 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into d99baeb997b04118ac98daacf8ad7d22(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:45,731 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:45,731 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=13, startTime=1681703745676; duration=0sec 2023-04-17 03:55:45,731 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:45,731 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:45,732 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:45,732 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:45,732 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,732 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/372a09ae991c4a4e88fec27fb966599d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=36.3 K 2023-04-17 03:55:45,733 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 372a09ae991c4a4e88fec27fb966599d, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1681703743108 2023-04-17 03:55:45,733 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 197590dc87984074a1104b133038bedb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=453, earliestPutTs=1681703743803 2023-04-17 03:55:45,733 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 84312d5ee97f43e597fdf626bf2e0d2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744592 2023-04-17 03:55:45,738 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:45,771 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,774 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:45,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,777 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,777 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,777 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,778 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,779 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,780 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1954a587[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,783 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,783 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,784 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,785 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,787 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,787 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,789 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,791 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,799 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,800 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,800 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703805815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703805816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703805816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703805816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703805824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,900 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:45,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:45,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:45,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:45,903 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:45,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:45,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:45,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,905 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,905 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:45,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:45,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:45,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:45,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:45,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:45,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:45,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:45,908 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:45,908 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:45,910 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,910 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:45,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@702ad099[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:45,911 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:45,912 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,913 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:45,913 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,913 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:45,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:45,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:45,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:45,923 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:45,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:45,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:45,923 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:45,924 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:45,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,925 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:45,926 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:45,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:45,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:45,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:45,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:45,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:45,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:45,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:45,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:45,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:45,930 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:45,931 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:45,931 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:45,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703806022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703806022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703806023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703806023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,031 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,036 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,036 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,037 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,037 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,038 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703806030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,039 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,039 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,039 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,039 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:46,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,042 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,042 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:46,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5659dc61[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:46,044 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:46,044 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,045 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:46,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:46,047 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,047 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:46,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:46,049 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:46,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:46,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:46,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:46,049 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:46,051 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,051 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,051 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:46,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:46,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,057 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,057 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,058 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,058 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:46,058 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:46,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:46,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/227bf9c0fa03412f859601b621c49c50 2023-04-17 03:55:46,128 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1dbab3773c4049f6b39c84d5991781dd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1dbab3773c4049f6b39c84d5991781dd 2023-04-17 03:55:46,129 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f204d58e1efb44888ea2661fdfa74000 2023-04-17 03:55:46,133 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into 1dbab3773c4049f6b39c84d5991781dd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:46,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:46,133 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=13, startTime=1681703745676; duration=0sec 2023-04-17 03:55:46,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:46,150 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/c0e5000d106042e8956a9edcfe972c28 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0e5000d106042e8956a9edcfe972c28 2023-04-17 03:55:46,153 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into c0e5000d106042e8956a9edcfe972c28(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:46,153 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:46,153 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=13, startTime=1681703745676; duration=0sec 2023-04-17 03:55:46,153 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:46,158 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,161 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,163 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,164 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,164 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,166 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,167 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,167 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,167 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,167 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,167 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:46,167 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:46,167 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,167 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:46,167 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:46,170 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:46,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c67dcfe[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:46,172 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:46,172 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,173 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,174 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:46,174 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:46,174 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:46,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:46,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,176 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:46,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:46,181 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:46,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:46,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:46,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:46,181 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:46,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,188 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:46,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,189 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:46,189 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,189 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:46,189 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:46,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,289 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,292 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,293 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,293 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,293 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,293 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,300 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,304 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a33a68[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:46,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:46,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,307 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:46,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:46,307 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:46,308 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:46,311 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:46,311 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:46,311 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:46,311 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:46,311 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:46,317 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,317 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,317 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,318 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,319 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,321 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,321 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,321 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,322 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,321 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:46,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:46,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:46,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,322 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:46,322 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703806330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703806331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703806331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703806331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703806340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,422 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,425 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,425 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,427 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,427 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,429 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,429 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,429 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,429 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:46,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,431 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:46,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b98fd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:46,433 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:46,433 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,434 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:46,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:46,435 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:46,436 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,436 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:46,438 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:46,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:46,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:46,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:46,438 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:46,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,441 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:46,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,444 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,444 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:46,444 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:46,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/6c3788703c6d4330ab1bce851521db35 2023-04-17 03:55:46,545 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/227bf9c0fa03412f859601b621c49c50 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50 2023-04-17 03:55:46,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50, entries=200, sequenceid=631, filesize=14.3 K 2023-04-17 03:55:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,552 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,553 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,553 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f204d58e1efb44888ea2661fdfa74000 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000 2023-04-17 03:55:46,555 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,555 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,555 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,556 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,557 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000, entries=150, sequenceid=631, filesize=11.9 K 2023-04-17 03:55:46,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/6c3788703c6d4330ab1bce851521db35 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35 2023-04-17 03:55:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,563 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,563 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,564 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:46,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,566 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:46,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e89e68d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:46,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:46,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,570 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:46,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:46,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:46,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:46,573 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,575 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35, entries=150, sequenceid=631, filesize=11.9 K 2023-04-17 03:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,576 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~344.72 KB/352992, currentSize=60.38 KB/61830 for 82824514d3f4f39e9f4f0e8595870ef8 in 897ms, sequenceid=631, compaction requested=false 2023-04-17 03:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:46,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:46,576 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:46,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:46,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:46,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:46,577 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:46,579 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,579 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,579 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:46,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:46,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,582 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:46,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,584 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:46,584 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:46,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:46,584 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,684 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:46,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:46,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:46,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:46,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:46,688 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:46,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:46,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:46,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:46,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,690 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:46,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:46,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:46,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:46,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:46,692 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:46,692 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:46,692 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:46,692 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:55:46,692 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:46,692 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:46,693 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:46,693 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:46,693 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:46,693 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:46,704 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5eba8bf27a0b4d66a0af8239e6b6e906 2023-04-17 03:55:46,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:46,842 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:46,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703806860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703746855, max timestamp=1681703746866Num uniques 859; 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703746855, max timestamp=1681703746866Num uniques 859; 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703746855, max timestamp=1681703746866Num uniques 859; 2023-04-17 03:55:46,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:55:46,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703806900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703806901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703806907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703806908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:46,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:46,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703806969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703807009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703807011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703807013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703807014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,126 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/c3bb5b320e0247e0a2b4a5805493eea5 2023-04-17 03:55:47,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703807176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703807213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703807214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703807219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703807220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42036 deadline: 1681703807483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703807518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703807522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703807527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:47,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703807527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,537 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/fe8817078a3f44588be29c7af5350220 2023-04-17 03:55:47,542 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5eba8bf27a0b4d66a0af8239e6b6e906 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906 2023-04-17 03:55:47,547 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906, entries=150, sequenceid=646, filesize=11.9 K 2023-04-17 03:55:47,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/c3bb5b320e0247e0a2b4a5805493eea5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5 2023-04-17 03:55:47,551 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5, entries=150, sequenceid=646, filesize=11.9 K 2023-04-17 03:55:47,552 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/fe8817078a3f44588be29c7af5350220 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220 2023-04-17 03:55:47,555 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220, entries=150, sequenceid=646, filesize=11.9 K 2023-04-17 03:55:47,556 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 82824514d3f4f39e9f4f0e8595870ef8 in 864ms, sequenceid=646, compaction requested=true 2023-04-17 03:55:47,556 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:47,556 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:47,556 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:47,561 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,562 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2b598e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:47,563 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:47,563 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,567 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:47,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:47,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:47,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:47,571 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:47,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:47,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:47,571 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:47,571 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:47,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,574 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,578 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:47,578 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,578 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,678 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:47,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:47,682 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:47,683 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,683 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,684 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:47,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:47,688 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:47,688 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:47,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:47,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:47,688 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-17 03:55:47,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:47,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:47,689 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:47,689 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:47,689 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:47,689 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:47,708 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/44f83870249e495e9e6b4e7450f920c4 2023-04-17 03:55:47,723 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/583b351e7bde482f861a8cc6458e2f64 2023-04-17 03:55:47,745 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/cacdc51bc2034918a29d5ee600c7d649 2023-04-17 03:55:47,749 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/44f83870249e495e9e6b4e7450f920c4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4 2023-04-17 03:55:47,756 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4, entries=150, sequenceid=679, filesize=11.9 K 2023-04-17 03:55:47,756 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/583b351e7bde482f861a8cc6458e2f64 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64 2023-04-17 03:55:47,760 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64, entries=150, sequenceid=679, filesize=11.9 K 2023-04-17 03:55:47,762 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/cacdc51bc2034918a29d5ee600c7d649 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649 2023-04-17 03:55:47,769 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649, entries=150, sequenceid=679, filesize=11.9 K 2023-04-17 03:55:47,769 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=0 B/0 for 82824514d3f4f39e9f4f0e8595870ef8 in 81ms, sequenceid=679, compaction requested=true 2023-04-17 03:55:47,770 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:47,770 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:47,771 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:47,772 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:47,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e24419[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:47,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:47,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,776 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:47,776 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:47,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,776 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:47,777 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,777 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,778 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:47,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:47,781 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:47,781 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:47,783 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,783 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,783 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,784 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,787 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,790 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:47,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:47,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:47,790 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,790 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:47,791 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,891 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:47,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:47,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:47,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:47,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:47,894 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:47,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:47,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:47,894 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:47,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,896 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,896 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:47,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:47,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:47,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:47,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:47,898 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:47,898 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:47,898 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:47,898 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,900 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,900 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:47,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd4d828[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:47,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:47,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,904 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:47,904 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:47,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:47,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,907 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:47,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:47,911 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:47,911 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:47,911 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:47,911 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:47,912 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:47,914 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,914 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:47,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,914 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:47,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:47,916 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:47,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:47,920 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:47,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:47,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:47,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,920 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:47,920 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:47,920 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:48,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:48,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,020 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/a005058f20544c8c941851763fdc679f 2023-04-17 03:55:48,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,025 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,026 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,027 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,027 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,028 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,028 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,028 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,029 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,029 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,031 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,032 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/941803d55e614ae49fd4149dcaa2f25c 2023-04-17 03:55:48,032 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eb8f0d1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,035 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,036 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,041 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,041 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,045 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,046 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,048 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,049 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,049 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/5978865ad2824739bb21466b186dbc82 2023-04-17 03:55:48,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/a005058f20544c8c941851763fdc679f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f 2023-04-17 03:55:48,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f, entries=150, sequenceid=690, filesize=11.9 K 2023-04-17 03:55:48,058 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/941803d55e614ae49fd4149dcaa2f25c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c 2023-04-17 03:55:48,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c, entries=150, sequenceid=690, filesize=11.9 K 2023-04-17 03:55:48,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/5978865ad2824739bb21466b186dbc82 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82 2023-04-17 03:55:48,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82, entries=150, sequenceid=690, filesize=11.9 K 2023-04-17 03:55:48,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 82824514d3f4f39e9f4f0e8595870ef8 in 50ms, sequenceid=690, compaction requested=true 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:48,066 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:48,066 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:48,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:48,066 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:48,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:48,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,067 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:48,067 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:48,068 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:48,068 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:48,068 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:48,068 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,068 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1dbab3773c4049f6b39c84d5991781dd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=60.2 K 2023-04-17 03:55:48,068 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:48,069 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,069 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d99baeb997b04118ac98daacf8ad7d22, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=62.6 K 2023-04-17 03:55:48,072 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1dbab3773c4049f6b39c84d5991781dd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744592 2023-04-17 03:55:48,072 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d99baeb997b04118ac98daacf8ad7d22, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744592 2023-04-17 03:55:48,072 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f204d58e1efb44888ea2661fdfa74000, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681703745653 2023-04-17 03:55:48,073 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 227bf9c0fa03412f859601b621c49c50, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681703745653 2023-04-17 03:55:48,073 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c3bb5b320e0247e0a2b4a5805493eea5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681703745702 2023-04-17 03:55:48,073 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5eba8bf27a0b4d66a0af8239e6b6e906, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681703745702 2023-04-17 03:55:48,073 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 583b351e7bde482f861a8cc6458e2f64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681703746899 2023-04-17 03:55:48,085 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 44f83870249e495e9e6b4e7450f920c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681703746899 2023-04-17 03:55:48,085 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 941803d55e614ae49fd4149dcaa2f25c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:48,085 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a005058f20544c8c941851763fdc679f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:48,095 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#451 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:48,102 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#452 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703808090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703808091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703808093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,103 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5baac366e1f5479cbcccfe2669a0b408 2023-04-17 03:55:48,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703808095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42028 deadline: 1681703808102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,123 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/d2e5c868fc784687a2f1140d3d38efed as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d2e5c868fc784687a2f1140d3d38efed 2023-04-17 03:55:48,124 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/cb36a1802ecb47eea1c7735cb5d8f923 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/cb36a1802ecb47eea1c7735cb5d8f923 2023-04-17 03:55:48,128 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into d2e5c868fc784687a2f1140d3d38efed(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:48,128 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:48,128 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=11, startTime=1681703748066; duration=0sec 2023-04-17 03:55:48,128 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into cb36a1802ecb47eea1c7735cb5d8f923(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:48,128 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:48,128 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:48,128 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:55:48,128 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=11, startTime=1681703748066; duration=0sec 2023-04-17 03:55:48,128 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:48,132 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:55:48,132 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:48,132 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,132 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0e5000d106042e8956a9edcfe972c28, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=60.2 K 2023-04-17 03:55:48,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c0e5000d106042e8956a9edcfe972c28, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681703744592 2023-04-17 03:55:48,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 6c3788703c6d4330ab1bce851521db35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1681703745653 2023-04-17 03:55:48,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fe8817078a3f44588be29c7af5350220, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681703745702 2023-04-17 03:55:48,133 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cacdc51bc2034918a29d5ee600c7d649, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1681703746899 2023-04-17 03:55:48,134 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5978865ad2824739bb21466b186dbc82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:48,141 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#454 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:48,149 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,149 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/bf36b196bd79486682169098e8a5ef9e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bf36b196bd79486682169098e8a5ef9e 2023-04-17 03:55:48,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,154 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,155 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,156 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,156 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,157 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into bf36b196bd79486682169098e8a5ef9e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:48,157 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:48,157 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=11, startTime=1681703748066; duration=0sec 2023-04-17 03:55:48,157 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:48,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,161 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,161 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,161 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,161 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,163 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b93fa1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,170 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,171 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,171 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,174 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,174 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,174 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,176 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,180 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,184 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,184 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,184 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703808203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703808203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703808204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703808206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703808210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,285 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,288 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,288 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,291 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,295 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6c6c23[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,296 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,297 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,297 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,298 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,298 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,299 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,299 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,301 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,302 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,306 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,308 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,310 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,310 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,310 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,310 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703808406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703808406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703808406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703808410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,411 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,414 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,414 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,415 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,416 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703808413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,418 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,418 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,418 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,418 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,418 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,419 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,420 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@600fe7e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,422 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,425 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,425 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,425 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,425 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,428 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,428 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,432 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,432 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,433 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,433 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,438 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,438 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,439 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,439 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,439 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1c34c4c4b32a4727bfd062be76f02672 2023-04-17 03:55:48,539 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,542 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,543 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,544 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,544 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,544 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,546 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,546 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,546 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,546 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,548 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ff7a817[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,550 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,551 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,558 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,558 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,558 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,559 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,559 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,560 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,560 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,564 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,564 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,564 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,664 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,668 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,668 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,669 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,671 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,671 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,671 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,671 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,671 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,672 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e92d27f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,676 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,677 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,677 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,677 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,677 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,677 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,677 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,678 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,679 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,682 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,682 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,682 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,683 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,684 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,684 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,685 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,688 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,689 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,689 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,689 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,689 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,689 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703808711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703808711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703808713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703808713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:48,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703808717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,789 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,792 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,792 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,793 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,794 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,794 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,795 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,795 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,795 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,796 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,796 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,796 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,796 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,798 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,798 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,798 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,798 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7203a6fa[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,800 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,801 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,802 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,803 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,805 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,809 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,809 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,811 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,811 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,811 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,812 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,815 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,820 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,820 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,821 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,921 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:48,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:48,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:48,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:48,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:48,924 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:48,924 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,924 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:48,924 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:48,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,927 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:48,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:48,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:48,929 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,929 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:48,929 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:48,929 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:48,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:48,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@737145ca[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:48,933 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:48,933 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=710 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/556fb8265b234c7db1e6aec91b0d0705 2023-04-17 03:55:48,934 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:48,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,937 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:48,937 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/5baac366e1f5479cbcccfe2669a0b408 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408 2023-04-17 03:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:48,940 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:48,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:48,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:48,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:48,940 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:48,942 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:48,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,944 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408, entries=300, sequenceid=710, filesize=19.0 K 2023-04-17 03:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,945 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1c34c4c4b32a4727bfd062be76f02672 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672 2023-04-17 03:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:48,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:48,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:48,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:48,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:48,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672, entries=150, sequenceid=710, filesize=11.9 K 2023-04-17 03:55:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,948 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:48,948 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:48,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:48,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:48,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:48,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/556fb8265b234c7db1e6aec91b0d0705 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705 2023-04-17 03:55:48,949 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:48,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:48,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705, entries=150, sequenceid=710, filesize=11.9 K 2023-04-17 03:55:48,953 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 82824514d3f4f39e9f4f0e8595870ef8 in 887ms, sequenceid=710, compaction requested=false 2023-04-17 03:55:48,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:49,049 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,052 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,052 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,053 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,056 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,056 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,056 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,056 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:55:49,056 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:49,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:49,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:49,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,073 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/d7a249f8c11541e9bf4d93b0af30df06 2023-04-17 03:55:49,093 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/0ee29a7622744cd49250cfafaf61e287 2023-04-17 03:55:49,103 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/3a4191cd8cd24bd0ad3399d77172ba1b 2023-04-17 03:55:49,108 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/d7a249f8c11541e9bf4d93b0af30df06 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06 2023-04-17 03:55:49,112 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06, entries=150, sequenceid=725, filesize=11.9 K 2023-04-17 03:55:49,113 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/0ee29a7622744cd49250cfafaf61e287 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287 2023-04-17 03:55:49,116 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287, entries=150, sequenceid=725, filesize=11.9 K 2023-04-17 03:55:49,117 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/3a4191cd8cd24bd0ad3399d77172ba1b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b 2023-04-17 03:55:49,122 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b, entries=150, sequenceid=725, filesize=11.9 K 2023-04-17 03:55:49,123 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 82824514d3f4f39e9f4f0e8595870ef8 in 67ms, sequenceid=725, compaction requested=true 2023-04-17 03:55:49,123 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:49,123 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:49,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,126 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:49,128 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:49,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134eca3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:49,128 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:49,130 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:49,131 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:49,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,131 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,132 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,132 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:49,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:49,136 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:49,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:49,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:49,136 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:49,136 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:49,138 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,138 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,138 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,141 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,144 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:49,144 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:49,144 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:49,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:49,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:49,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:49,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/749a0ef79c834ec695cfceda3f8d3089 2023-04-17 03:55:49,244 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/977fab211c1f4c76a3ec718b9c8ffc77 2023-04-17 03:55:49,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,248 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:49,256 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,258 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cdd7d0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:49,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:49,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,261 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:49,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,261 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:49,262 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,262 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:49,264 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:49,264 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:49,267 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,267 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,267 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,273 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,274 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:49,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:49,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,274 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:49,274 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:49,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,274 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703749244, max timestamp=1681703749286Num uniques 859; 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703749244, max timestamp=1681703749286Num uniques 859; 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681703749244, max timestamp=1681703749286Num uniques 859; 2023-04-17 03:55:49,288 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-17 03:55:49,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703809309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703809310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703809310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703809311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703809312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,374 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,377 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:49,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:49,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:49,384 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,384 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16169a65[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,390 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:49,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:49,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:49,393 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:49,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:49,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:49,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:49,393 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:49,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,396 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,398 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:49,401 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:49,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,403 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:49,403 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703809420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703809421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703809421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703809421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703809422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,503 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,506 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,507 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,509 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,510 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,510 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,510 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:49,510 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:49,511 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb0c278[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:49,513 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:49,513 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,514 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:49,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:49,517 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:49,519 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:49,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:49,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:49,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:49,520 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,522 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,523 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,526 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:49,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,526 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:49,526 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:49,626 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,630 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,630 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,631 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,634 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,634 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,635 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,635 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:49,635 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:49,635 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:49,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703809631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703809631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703809631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:49,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703809631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703809631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,638 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c76ad9[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:49,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:49,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:49,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,643 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:49,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,645 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:49,645 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,645 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:49,647 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:49,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:49,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:49,647 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:49,647 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:49,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:49,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,650 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,653 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,656 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:49,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:49,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,657 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:49,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:49,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:49,657 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:49,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,658 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:49,658 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:49,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b7b66a6c4d4445bf8fd987175baa67ec 2023-04-17 03:55:49,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/749a0ef79c834ec695cfceda3f8d3089 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089 2023-04-17 03:55:49,675 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089, entries=250, sequenceid=736, filesize=16.7 K 2023-04-17 03:55:49,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/977fab211c1f4c76a3ec718b9c8ffc77 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77 2023-04-17 03:55:49,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77, entries=150, sequenceid=736, filesize=11.9 K 2023-04-17 03:55:49,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b7b66a6c4d4445bf8fd987175baa67ec as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec 2023-04-17 03:55:49,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec, entries=150, sequenceid=736, filesize=11.9 K 2023-04-17 03:55:49,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 82824514d3f4f39e9f4f0e8595870ef8 in 457ms, sequenceid=736, compaction requested=true 2023-04-17 03:55:49,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:49,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:49,684 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:49,684 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:49,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:49,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 61864 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:49,685 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,685 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,685 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/cb36a1802ecb47eea1c7735cb5d8f923, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=60.4 K 2023-04-17 03:55:49,685 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d2e5c868fc784687a2f1140d3d38efed, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.5 K 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting cb36a1802ecb47eea1c7735cb5d8f923, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:49,685 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d2e5c868fc784687a2f1140d3d38efed, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:49,686 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 5baac366e1f5479cbcccfe2669a0b408, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681703748040 2023-04-17 03:55:49,686 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1c34c4c4b32a4727bfd062be76f02672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681703748040 2023-04-17 03:55:49,686 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting d7a249f8c11541e9bf4d93b0af30df06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681703748089 2023-04-17 03:55:49,686 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0ee29a7622744cd49250cfafaf61e287, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681703748089 2023-04-17 03:55:49,687 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 749a0ef79c834ec695cfceda3f8d3089, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:49,687 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 977fab211c1f4c76a3ec718b9c8ffc77, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:49,695 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#462 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:49,698 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#463 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:49,710 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/7b13005edd944d10a70a6ccfbb94e44b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7b13005edd944d10a70a6ccfbb94e44b 2023-04-17 03:55:49,714 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into 7b13005edd944d10a70a6ccfbb94e44b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:49,714 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:49,714 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=12, startTime=1681703749684; duration=0sec 2023-04-17 03:55:49,714 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:49,714 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:49,715 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:49,715 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:49,715 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,715 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bf36b196bd79486682169098e8a5ef9e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.5 K 2023-04-17 03:55:49,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting bf36b196bd79486682169098e8a5ef9e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1681703747998 2023-04-17 03:55:49,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 556fb8265b234c7db1e6aec91b0d0705, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=710, earliestPutTs=1681703748040 2023-04-17 03:55:49,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 3a4191cd8cd24bd0ad3399d77172ba1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=725, earliestPutTs=1681703748089 2023-04-17 03:55:49,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b7b66a6c4d4445bf8fd987175baa67ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:49,724 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:49,758 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:49,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:49,761 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:49,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:49,761 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:49,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,762 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,763 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,763 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:49,764 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,764 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:49,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:49,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:49,766 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:49,766 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:49,766 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-17 03:55:49,766 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:49,767 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:49,767 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:49,767 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:49,767 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:49,767 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:49,773 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6468a92f72fb413e8fa2e9da545c70c5 2023-04-17 03:55:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:49,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:49,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703809945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703809946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703809951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703809952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:49,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:49,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703809953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703810053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703810053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703810054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703810059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703810060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,110 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/4d41eaebf9e34d96bc8023027aaab3b0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/4d41eaebf9e34d96bc8023027aaab3b0 2023-04-17 03:55:50,114 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into 4d41eaebf9e34d96bc8023027aaab3b0(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:50,114 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:50,114 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=12, startTime=1681703749684; duration=0sec 2023-04-17 03:55:50,114 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:50,134 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/4d787af6cafc4d6a9795c5635ab5c6fb as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/4d787af6cafc4d6a9795c5635ab5c6fb 2023-04-17 03:55:50,138 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into 4d787af6cafc4d6a9795c5635ab5c6fb(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:50,138 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:50,138 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=12, startTime=1681703749684; duration=0sec 2023-04-17 03:55:50,138 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:50,186 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f80c31e999a2467a82c80ae5ef47fd16 2023-04-17 03:55:50,196 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=771 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/097da474899b4f9d9cd99a4d883c4c05 2023-04-17 03:55:50,201 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6468a92f72fb413e8fa2e9da545c70c5 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5 2023-04-17 03:55:50,205 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5, entries=150, sequenceid=771, filesize=11.9 K 2023-04-17 03:55:50,206 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/f80c31e999a2467a82c80ae5ef47fd16 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16 2023-04-17 03:55:50,209 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16, entries=150, sequenceid=771, filesize=11.9 K 2023-04-17 03:55:50,210 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/097da474899b4f9d9cd99a4d883c4c05 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05 2023-04-17 03:55:50,213 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05, entries=150, sequenceid=771, filesize=11.9 K 2023-04-17 03:55:50,214 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=60.38 KB/61830 for 82824514d3f4f39e9f4f0e8595870ef8 in 448ms, sequenceid=771, compaction requested=false 2023-04-17 03:55:50,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:50,214 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:50,214 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:50,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,218 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,219 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:50,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ea34766[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:50,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:50,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:50,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:50,223 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:50,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:50,224 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,224 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:50,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:50,227 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:50,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:50,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:50,227 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:50,228 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:50,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,229 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,230 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,231 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,234 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:50,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:50,235 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:50,235 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:50,235 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,235 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:50,267 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:50,269 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/2dd7c7b3fef44234bf7124e86f719533 2023-04-17 03:55:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703810286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703810287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703810288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703810289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703810290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,335 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:50,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:50,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:50,338 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,341 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:50,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:50,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:50,342 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:50,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:50,344 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:50,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c443b65[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:50,345 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:50,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:50,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:50,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:50,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:50,348 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,348 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:50,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:50,352 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:50,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:50,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:50,352 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:50,352 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:50,354 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,354 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,355 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,356 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:50,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,359 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:50,358 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,359 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,359 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:50,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:50,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703810394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703810394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703810395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703810395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703810396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,459 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:50,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:50,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:50,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:50,463 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:50,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:50,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:50,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,467 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,467 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:50,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:50,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:50,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:50,471 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,471 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150e9597[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:50,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:50,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,476 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:50,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:50,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:50,481 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:50,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:50,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:50,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:50,481 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:50,482 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,485 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:50,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:50,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,488 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:50,488 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:50,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,488 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,588 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:50,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:50,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:50,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:50,591 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:50,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:50,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:50,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,593 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,593 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,594 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:50,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:50,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:50,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:50,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:50,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:50,595 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:50,595 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:50,596 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,596 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67edb44d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:50,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:50,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,602 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:50,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,602 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:50,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:50,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703810599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703810600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,604 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:50,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703810603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:50,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:50,608 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:50,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:50,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:50,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:50,608 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:50,609 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,609 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,610 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,610 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703810605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703810605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,613 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:50,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:50,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:50,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:50,616 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,616 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:50,616 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:50,616 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:50,616 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:50,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/963e6a03777141179b3106f8d5f83e13 2023-04-17 03:55:50,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=789 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/dd1b8980cff84affb4b377efd50fe060 2023-04-17 03:55:50,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/2dd7c7b3fef44234bf7124e86f719533 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533 2023-04-17 03:55:50,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533, entries=250, sequenceid=789, filesize=16.7 K 2023-04-17 03:55:50,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/963e6a03777141179b3106f8d5f83e13 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13 2023-04-17 03:55:50,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13, entries=150, sequenceid=789, filesize=11.9 K 2023-04-17 03:55:50,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/dd1b8980cff84affb4b377efd50fe060 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060 2023-04-17 03:55:50,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060, entries=150, sequenceid=789, filesize=11.9 K 2023-04-17 03:55:50,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 82824514d3f4f39e9f4f0e8595870ef8 in 448ms, sequenceid=789, compaction requested=true 2023-04-17 03:55:50,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:50,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:50,715 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:50,715 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:50,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:50,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:50,716 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:50,716 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:50,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42494 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:50,716 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:50,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:50,717 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,717 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7b13005edd944d10a70a6ccfbb94e44b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=41.5 K 2023-04-17 03:55:50,716 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:50,717 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,717 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/4d41eaebf9e34d96bc8023027aaab3b0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=36.7 K 2023-04-17 03:55:50,717 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 7b13005edd944d10a70a6ccfbb94e44b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:50,717 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4d41eaebf9e34d96bc8023027aaab3b0, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:50,718 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6468a92f72fb413e8fa2e9da545c70c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681703749302 2023-04-17 03:55:50,718 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting f80c31e999a2467a82c80ae5ef47fd16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681703749302 2023-04-17 03:55:50,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:50,719 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:50,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:50,719 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:50,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,721 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,722 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:50,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:50,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:50,723 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 963e6a03777141179b3106f8d5f83e13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:50,723 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2dd7c7b3fef44234bf7124e86f719533, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:50,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:50,725 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:50,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:50,726 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,726 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:50,726 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 03:55:50,730 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#472 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:50,730 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#471 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:50,732 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:50,764 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/d75ffc95b9ef427db1db3f0a31cffeb3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d75ffc95b9ef427db1db3f0a31cffeb3 2023-04-17 03:55:50,772 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into d75ffc95b9ef427db1db3f0a31cffeb3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:50,772 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:50,772 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=13, startTime=1681703750715; duration=0sec 2023-04-17 03:55:50,773 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:50,773 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:50,774 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:50,774 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:50,774 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:50,774 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/4d787af6cafc4d6a9795c5635ab5c6fb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=36.7 K 2023-04-17 03:55:50,774 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4d787af6cafc4d6a9795c5635ab5c6fb, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681703749218 2023-04-17 03:55:50,775 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 097da474899b4f9d9cd99a4d883c4c05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=771, earliestPutTs=1681703749302 2023-04-17 03:55:50,775 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting dd1b8980cff84affb4b377efd50fe060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:50,783 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:50,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:50,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703810922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703810925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703810928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703810929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:50,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:50,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703810930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703811031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703811034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703811039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703811039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703811040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,159 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/df3318dda3cc4c09a86a3536913f2623 2023-04-17 03:55:51,163 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/e889aeeb40fa44da831b80da62d7815a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e889aeeb40fa44da831b80da62d7815a 2023-04-17 03:55:51,170 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into e889aeeb40fa44da831b80da62d7815a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:51,170 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:51,170 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=13, startTime=1681703750715; duration=0sec 2023-04-17 03:55:51,171 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:51,193 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/1f01042632244e2a9e8076e0a30af510 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/1f01042632244e2a9e8076e0a30af510 2023-04-17 03:55:51,200 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into 1f01042632244e2a9e8076e0a30af510(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:51,200 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:51,200 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=13, startTime=1681703750716; duration=0sec 2023-04-17 03:55:51,200 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:51,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703811236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703811239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703811244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703811245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703811245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703811542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703811542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703811550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703811551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:51,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703811551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,574 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/2d98d84f81794cf0bcc1f762fa418195 2023-04-17 03:55:51,586 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/2b55bebe5dd84395a97c1d83466be3b3 2023-04-17 03:55:51,590 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/df3318dda3cc4c09a86a3536913f2623 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623 2023-04-17 03:55:51,596 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623, entries=150, sequenceid=805, filesize=11.9 K 2023-04-17 03:55:51,597 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/2d98d84f81794cf0bcc1f762fa418195 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195 2023-04-17 03:55:51,601 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195, entries=150, sequenceid=805, filesize=11.9 K 2023-04-17 03:55:51,601 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/2b55bebe5dd84395a97c1d83466be3b3 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3 2023-04-17 03:55:51,620 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3, entries=150, sequenceid=805, filesize=11.9 K 2023-04-17 03:55:51,621 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 82824514d3f4f39e9f4f0e8595870ef8 in 895ms, sequenceid=805, compaction requested=false 2023-04-17 03:55:51,621 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:51,621 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,623 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@233147a1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:51,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:51,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:51,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:51,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:51,629 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:51,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:51,633 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:51,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:51,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:51,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:51,633 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:51,634 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,635 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:51,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,635 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,636 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,641 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,641 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:51,641 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:51,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:51,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:51,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:51,641 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:51,641 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:51,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:51,642 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:51,642 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:51,642 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:51,742 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:51,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:51,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:51,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:51,745 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:51,745 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:51,745 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:51,745 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:51,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,746 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:51,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:51,747 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:51,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:51,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:51,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:51,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:51,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:51,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:51,750 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-17 03:55:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:51,751 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:51,751 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:51,751 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:51,751 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:51,751 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:51,767 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1cadc5f60fdf473a8e669f3589d4aa7f 2023-04-17 03:55:52,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:52,051 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:52,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703812066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703812066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703812067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703812068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703812070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,177 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/377ef43dfde049bc93bf2b0769e072a9 2023-04-17 03:55:52,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703812176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703812176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703812176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703812177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703812178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703812385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703812385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703812385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703812386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703812388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,587 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=825 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/513a39360204489e9553be20689c9113 2023-04-17 03:55:52,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/1cadc5f60fdf473a8e669f3589d4aa7f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f 2023-04-17 03:55:52,595 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f, entries=150, sequenceid=825, filesize=11.9 K 2023-04-17 03:55:52,596 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/377ef43dfde049bc93bf2b0769e072a9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9 2023-04-17 03:55:52,599 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9, entries=150, sequenceid=825, filesize=11.9 K 2023-04-17 03:55:52,599 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/513a39360204489e9553be20689c9113 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113 2023-04-17 03:55:52,603 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113, entries=150, sequenceid=825, filesize=11.9 K 2023-04-17 03:55:52,603 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 82824514d3f4f39e9f4f0e8595870ef8 in 853ms, sequenceid=825, compaction requested=true 2023-04-17 03:55:52,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:52,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:52,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,605 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef3dfb1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:52,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:52,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,609 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,610 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:52,610 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:52,610 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:52,610 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,610 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:52,612 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,612 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:52,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:52,615 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:52,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:52,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:52,615 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:52,615 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:52,616 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,616 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,617 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,621 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,621 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,622 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:52,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,622 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,622 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:52,622 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:52,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:52,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 03:55:52,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:52,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:52,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:52,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:52,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:52,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:52,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703812711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703812711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703812712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703812713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703812718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,722 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:52,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:52,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:52,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:52,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:52,726 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:52,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:52,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:52,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,727 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,727 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,728 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,728 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:52,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:52,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:52,729 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:52,729 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:52,730 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:52,732 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,732 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:52,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb04fd5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:52,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:52,735 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,736 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:52,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:52,737 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:52,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,737 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:52,741 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,741 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:52,743 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:52,743 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:52,743 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:52,744 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:52,744 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:52,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,746 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,748 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,752 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,752 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:52,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,753 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,753 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:52,753 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:52,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703812819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703812819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703812821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703812821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:52,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703812822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,853 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:52,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:52,857 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:52,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:52,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:52,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,857 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,859 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,859 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,859 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,859 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:52,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:52,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:52,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:52,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:52,861 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:52,863 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,863 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:52,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1575dfb2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:52,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:52,865 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,867 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:52,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,867 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:52,867 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:52,868 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,868 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:52,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:52,871 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:52,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:52,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:52,871 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:52,871 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:52,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,873 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,873 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,875 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,877 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:52,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:52,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,877 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:52,878 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:52,878 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:52,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:52,978 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:52,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:52,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:52,981 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:52,981 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:52,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:52,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,982 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,983 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:52,983 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:52,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,984 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:52,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:52,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:52,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:52,986 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:52,987 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,987 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:52,987 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:52,987 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:52,987 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,989 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,989 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:52,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c508ef2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:52,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:52,991 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,993 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:52,993 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:52,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,993 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:52,995 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:52,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:52,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:52,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:52,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:52,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:52,999 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:52,999 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,001 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,002 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,003 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,007 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,008 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,008 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,009 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,009 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703813024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703813025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703813026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703813028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/42b1268e598d42a289f29c84b6f33526 2023-04-17 03:55:53,109 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,114 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,114 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,116 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,116 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,117 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1a86013d41394417aef6ccce590ddd6f 2023-04-17 03:55:53,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:53,118 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,118 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,121 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,121 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,121 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,121 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c3ec09e[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,125 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,126 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,126 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,126 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,129 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,130 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,134 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,134 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,134 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,136 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,136 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,136 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,144 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,144 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,145 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,145 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,145 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,245 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,249 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,249 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,251 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,252 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,252 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,253 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,253 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:53,253 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,253 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,254 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@416df22b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,257 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,258 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,263 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,263 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,265 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,266 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,272 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,272 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:53,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703813335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703813335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703813335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703813335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703813335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,372 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,375 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,375 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,376 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,376 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,378 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,379 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,380 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:53,380 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,382 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ade3dfb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,384 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,385 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,386 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,386 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,386 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,387 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,388 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,392 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,396 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:53,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,399 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,399 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,399 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,400 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,500 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,507 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,507 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,508 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,509 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,511 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,511 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,511 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:53,511 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,511 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,513 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c10b4c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,515 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,516 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,522 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,524 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,524 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,526 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,527 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,528 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=841 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/62ea5bb7c74742489d91b327151bcd44 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,533 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,533 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,533 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,533 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/42b1268e598d42a289f29c84b6f33526 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526 2023-04-17 03:55:53,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526, entries=200, sequenceid=841, filesize=14.3 K 2023-04-17 03:55:53,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1a86013d41394417aef6ccce590ddd6f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f 2023-04-17 03:55:53,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f, entries=150, sequenceid=841, filesize=11.9 K 2023-04-17 03:55:53,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/62ea5bb7c74742489d91b327151bcd44 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44 2023-04-17 03:55:53,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44, entries=150, sequenceid=841, filesize=11.9 K 2023-04-17 03:55:53,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 82824514d3f4f39e9f4f0e8595870ef8 in 860ms, sequenceid=841, compaction requested=true 2023-04-17 03:55:53,552 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:53,553 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:53,553 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:53,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:53,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:53,554 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52342 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:53,554 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:53,554 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,554 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e889aeeb40fa44da831b80da62d7815a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=51.1 K 2023-04-17 03:55:53,554 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:53,555 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:53,555 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting e889aeeb40fa44da831b80da62d7815a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:53,555 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,555 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d75ffc95b9ef427db1db3f0a31cffeb3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.7 K 2023-04-17 03:55:53,555 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting df3318dda3cc4c09a86a3536913f2623, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1681703750287 2023-04-17 03:55:53,555 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d75ffc95b9ef427db1db3f0a31cffeb3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:53,555 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 1cadc5f60fdf473a8e669f3589d4aa7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=825, earliestPutTs=1681703750927 2023-04-17 03:55:53,556 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2d98d84f81794cf0bcc1f762fa418195, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1681703750287 2023-04-17 03:55:53,556 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 42b1268e598d42a289f29c84b6f33526, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752063 2023-04-17 03:55:53,556 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 377ef43dfde049bc93bf2b0769e072a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=825, earliestPutTs=1681703750927 2023-04-17 03:55:53,556 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1a86013d41394417aef6ccce590ddd6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752067 2023-04-17 03:55:53,563 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:53,565 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:53,577 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/e3444968122448008122023221efbf84 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e3444968122448008122023221efbf84 2023-04-17 03:55:53,581 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into e3444968122448008122023221efbf84(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:53,581 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,581 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=12, startTime=1681703753553; duration=0sec 2023-04-17 03:55:53,581 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:53,582 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-17 03:55:53,582 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-17 03:55:53,583 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:53,583 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,583 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/1f01042632244e2a9e8076e0a30af510, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=48.7 K 2023-04-17 03:55:53,584 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1f01042632244e2a9e8076e0a30af510, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=789, earliestPutTs=1681703749942 2023-04-17 03:55:53,584 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2b55bebe5dd84395a97c1d83466be3b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1681703750287 2023-04-17 03:55:53,584 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 513a39360204489e9553be20689c9113, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=825, earliestPutTs=1681703750927 2023-04-17 03:55:53,585 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 62ea5bb7c74742489d91b327151bcd44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752067 2023-04-17 03:55:53,593 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:53,601 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/cbff4a1029ee47d4ad69b6c0d2ec8e26 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cbff4a1029ee47d4ad69b6c0d2ec8e26 2023-04-17 03:55:53,604 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into cbff4a1029ee47d4ad69b6c0d2ec8e26(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:53,604 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,604 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=12, startTime=1681703753553; duration=0sec 2023-04-17 03:55:53,604 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:53,633 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,636 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,639 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,641 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,641 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,641 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:53,642 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,652 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/53a6596aba3048988a4c8d3aca18c067 2023-04-17 03:55:53,665 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/195a5b6bfe66410d90623b33eb5e9942 2023-04-17 03:55:53,679 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/da661952077746989a7805271d5d7012 2023-04-17 03:55:53,684 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/53a6596aba3048988a4c8d3aca18c067 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067 2023-04-17 03:55:53,687 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067, entries=150, sequenceid=859, filesize=11.9 K 2023-04-17 03:55:53,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/195a5b6bfe66410d90623b33eb5e9942 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942 2023-04-17 03:55:53,692 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942, entries=150, sequenceid=859, filesize=11.9 K 2023-04-17 03:55:53,693 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/da661952077746989a7805271d5d7012 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012 2023-04-17 03:55:53,696 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012, entries=150, sequenceid=859, filesize=11.9 K 2023-04-17 03:55:53,696 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 82824514d3f4f39e9f4f0e8595870ef8 in 55ms, sequenceid=859, compaction requested=false 2023-04-17 03:55:53,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,696 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,698 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d109ba[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,700 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,702 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,703 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,703 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,706 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,706 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,706 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,709 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,710 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,711 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,715 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,715 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,716 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,716 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,716 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,716 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:53,816 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,820 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,821 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,821 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,822 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,822 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,824 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,824 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,824 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,824 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,826 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,826 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46035221[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,828 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,830 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,830 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,830 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,831 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,831 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,832 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,834 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,834 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,835 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,840 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,840 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,840 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,840 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:53,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:53,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:53,859 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/87199aaeaf4c46548808a51971e0735d 2023-04-17 03:55:53,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703813877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703813878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703813879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703753875, max timestamp=1681703753884Num uniques 859; 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703753875, max timestamp=1681703753884Num uniques 859; 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703753875, max timestamp=1681703753884Num uniques 859; 2023-04-17 03:55:53,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:53,940 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:53,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:53,944 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:53,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:53,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:53,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:53,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:53,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:53,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:53,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:53,949 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:53,949 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:53,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703813942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703813942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:53,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:53,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4db98eff[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:53,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:53,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,954 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:53,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:53,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:53,956 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:53,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:53,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:53,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:53,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:53,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:53,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:53,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:53,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,960 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:53,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:53,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:53,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:53,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:53,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:53,965 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:53,965 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:53,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:53,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:53,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:53,972 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/a8816e10bd674abf8c0ed6730f7f7f3a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a8816e10bd674abf8c0ed6730f7f7f3a 2023-04-17 03:55:53,975 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into a8816e10bd674abf8c0ed6730f7f7f3a(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:53,975 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:53,975 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=12, startTime=1681703753553; duration=0sec 2023-04-17 03:55:53,975 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703813986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703813987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703813987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703814051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703814051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,065 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:54,069 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:54,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:54,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:54,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,074 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,074 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:54,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:54,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:54,076 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:54,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768676f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:54,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:54,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,082 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:54,082 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:54,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:54,084 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:54,086 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:54,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:54,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:54,086 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:54,086 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:54,088 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,096 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,101 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,101 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:54,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:54,101 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:54,101 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703814190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703814191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703814191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,201 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:54,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:54,205 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:54,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,205 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:54,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:54,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,207 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,208 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:54,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:54,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:54,209 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,209 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:54,209 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:54,209 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,209 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,211 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:54,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671432e1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:54,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:54,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,218 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:54,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:54,220 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,220 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:54,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:54,223 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:54,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:54,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:54,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:54,223 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:54,225 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,225 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,225 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,226 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,228 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,233 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,233 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,232 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:54,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:54,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,233 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:54,233 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:54,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703814256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703814257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/aabc6795ef7e49f79881a89d9365a2a6 2023-04-17 03:55:54,333 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:54,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:54,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:54,336 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,338 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,338 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,339 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:54,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:54,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:54,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:54,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:54,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:54,344 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:54,344 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:54,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,346 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,347 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:54,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bdaf678[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:54,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:54,348 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,349 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,349 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:54,350 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:54,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,351 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:54,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:54,355 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:54,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:54,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:54,355 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:54,355 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:54,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,356 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,357 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,358 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:54,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:54,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,366 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:54,366 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:54,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,367 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:54,467 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:54,470 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:54,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:54,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:54,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,471 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,473 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:54,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:54,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:54,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:54,475 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:54,475 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:54,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:54,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:54,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:54,477 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a15498a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:54,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:54,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:54,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:54,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:54,485 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,485 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:54,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:54,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:54,487 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:54,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:54,487 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:54,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:54,488 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:54,489 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,489 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,489 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,490 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,493 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,493 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,493 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:54,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,494 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:54,494 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:54,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42028 deadline: 1681703814494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42072 deadline: 1681703814498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42062 deadline: 1681703814502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42092 deadline: 1681703814563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42036 deadline: 1681703814563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,594 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:54,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:54,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:54,598 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,600 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,600 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,601 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,601 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:54,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:54,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:54,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:54,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:54,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:54,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:54,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:54,602 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:54,602 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,602 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. started... 2023-04-17 03:55:54,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. as already flushing 2023-04-17 03:55:54,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:54,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:54,604 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,604 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,605 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:54,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e8202b8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:54,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:54,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:54,608 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:54,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:54,609 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:54,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:54,612 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:54,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:54,612 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:54,613 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:54,613 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:54,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,615 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:54,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:54,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,617 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:54,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,619 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,619 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,619 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:54,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:54,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:54,619 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:54,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:54,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:54,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:54,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:54,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:54,620 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:54,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:54,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:54,620 DEBUG [Thread-2928] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:54,620 DEBUG [Thread-2928] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:54,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:54,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:54,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=871 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/dc604df01dbb41c89555cd6fcf3eb8ea 2023-04-17 03:55:54,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/87199aaeaf4c46548808a51971e0735d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d 2023-04-17 03:55:54,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d, entries=200, sequenceid=871, filesize=14.3 K 2023-04-17 03:55:54,687 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/aabc6795ef7e49f79881a89d9365a2a6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6 2023-04-17 03:55:54,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6, entries=150, sequenceid=871, filesize=11.9 K 2023-04-17 03:55:54,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/dc604df01dbb41c89555cd6fcf3eb8ea as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea 2023-04-17 03:55:54,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea, entries=150, sequenceid=871, filesize=11.9 K 2023-04-17 03:55:54,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 82824514d3f4f39e9f4f0e8595870ef8 in 846ms, sequenceid=871, compaction requested=true 2023-04-17 03:55:54,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:54,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:54,697 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:54,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:54,697 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:54,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/A is initiating minor compaction (all files) 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/B is initiating minor compaction (all files) 2023-04-17 03:55:54,698 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/A in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,698 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/B in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,698 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a8816e10bd674abf8c0ed6730f7f7f3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=39.3 K 2023-04-17 03:55:54,698 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e3444968122448008122023221efbf84, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=37.0 K 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e3444968122448008122023221efbf84, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752067 2023-04-17 03:55:54,698 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a8816e10bd674abf8c0ed6730f7f7f3a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752067 2023-04-17 03:55:54,699 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 195a5b6bfe66410d90623b33eb5e9942, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1681703752711 2023-04-17 03:55:54,699 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 53a6596aba3048988a4c8d3aca18c067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1681703752711 2023-04-17 03:55:54,699 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting aabc6795ef7e49f79881a89d9365a2a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681703753846 2023-04-17 03:55:54,699 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 87199aaeaf4c46548808a51971e0735d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681703753846 2023-04-17 03:55:54,705 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#B#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:54,705 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#A#compaction#493 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:54,712 DEBUG [Thread-2933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:60620 2023-04-17 03:55:54,712 DEBUG [Thread-2933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:54,713 DEBUG [Thread-2935] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:60620 2023-04-17 03:55:54,713 DEBUG [Thread-2935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:54,717 DEBUG [Thread-2931] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:60620 2023-04-17 03:55:54,717 DEBUG [Thread-2931] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:54,720 DEBUG [Thread-2937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:60620 2023-04-17 03:55:54,720 DEBUG [Thread-2937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:54,720 DEBUG [Thread-2929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:60620 2023-04-17 03:55:54,720 DEBUG [Thread-2929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:54,720 DEBUG [Thread-2928] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:54,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:54,722 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/ab5716418a9c4aa7b34e326d731ebcde as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/ab5716418a9c4aa7b34e326d731ebcde 2023-04-17 03:55:54,725 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/B of 82824514d3f4f39e9f4f0e8595870ef8 into ab5716418a9c4aa7b34e326d731ebcde(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:54,725 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:54,725 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/B, priority=13, startTime=1681703754697; duration=0sec 2023-04-17 03:55:54,725 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:55:54,725 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:55:54,726 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:55:54,726 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 82824514d3f4f39e9f4f0e8595870ef8/C is initiating minor compaction (all files) 2023-04-17 03:55:54,726 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 82824514d3f4f39e9f4f0e8595870ef8/C in TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:54,726 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cbff4a1029ee47d4ad69b6c0d2ec8e26, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp, totalSize=37.0 K 2023-04-17 03:55:54,727 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting cbff4a1029ee47d4ad69b6c0d2ec8e26, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=841, earliestPutTs=1681703752067 2023-04-17 03:55:54,727 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting da661952077746989a7805271d5d7012, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1681703752711 2023-04-17 03:55:54,727 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting dc604df01dbb41c89555cd6fcf3eb8ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=871, earliestPutTs=1681703753846 2023-04-17 03:55:54,731 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 82824514d3f4f39e9f4f0e8595870ef8#C#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:55:55,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:55,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-17 03:55:55,005 DEBUG [Thread-2920] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:60620 2023-04-17 03:55:55,005 DEBUG [Thread-2920] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:55,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-17 03:55:55,005 DEBUG [Thread-2924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:60620 2023-04-17 03:55:55,005 DEBUG [Thread-2924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,011 DEBUG [Thread-2918] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:60620 2023-04-17 03:55:55,011 DEBUG [Thread-2918] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0dfbfd75b0814c64b78ddfde9166aa0f 2023-04-17 03:55:55,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/fa79732383c347a0bdfc9f20b272d3cc 2023-04-17 03:55:55,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b0903c3206cc41ea98005005c5eb5e30 2023-04-17 03:55:55,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/0dfbfd75b0814c64b78ddfde9166aa0f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0dfbfd75b0814c64b78ddfde9166aa0f 2023-04-17 03:55:55,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0dfbfd75b0814c64b78ddfde9166aa0f, entries=150, sequenceid=911, filesize=11.9 K 2023-04-17 03:55:55,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/fa79732383c347a0bdfc9f20b272d3cc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fa79732383c347a0bdfc9f20b272d3cc 2023-04-17 03:55:55,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fa79732383c347a0bdfc9f20b272d3cc, entries=150, sequenceid=911, filesize=11.9 K 2023-04-17 03:55:55,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/b0903c3206cc41ea98005005c5eb5e30 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b0903c3206cc41ea98005005c5eb5e30 2023-04-17 03:55:55,043 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b0903c3206cc41ea98005005c5eb5e30, entries=150, sequenceid=911, filesize=11.9 K 2023-04-17 03:55:55,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=6.71 KB/6870 for 82824514d3f4f39e9f4f0e8595870ef8 in 39ms, sequenceid=911, compaction requested=false 2023-04-17 03:55:55,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:55,071 DEBUG [Thread-2926] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:60620 2023-04-17 03:55:55,071 DEBUG [Thread-2926] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,073 DEBUG [Thread-2922] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:60620 2023-04-17 03:55:55,073 DEBUG [Thread-2922] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 183 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 213 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 173 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1728 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5184 rows 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1738 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5211 rows 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1736 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5208 rows 2023-04-17 03:55:55,073 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1742 2023-04-17 03:55:55,074 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5222 rows 2023-04-17 03:55:55,074 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(402): scanned 1750 2023-04-17 03:55:55,074 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(403): verified 5249 rows 2023-04-17 03:55:55,074 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:55:55,074 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:60620 2023-04-17 03:55:55,074 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:55:55,075 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:55:55,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:55:55,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:55,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:55,081 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703755081"}]},"ts":"1681703755081"} 2023-04-17 03:55:55,082 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:55:55,084 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:55:55,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, UNASSIGN}] 2023-04-17 03:55:55,086 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, UNASSIGN 2023-04-17 03:55:55,086 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=82824514d3f4f39e9f4f0e8595870ef8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:55,086 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703755086"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703755086"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703755086"}]},"ts":"1681703755086"} 2023-04-17 03:55:55,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:55,121 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/e5f861ea01844bdcb44302ad4bf8db6f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e5f861ea01844bdcb44302ad4bf8db6f 2023-04-17 03:55:55,124 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/A of 82824514d3f4f39e9f4f0e8595870ef8 into e5f861ea01844bdcb44302ad4bf8db6f(size=13.3 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:55,125 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:55,125 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/A, priority=13, startTime=1681703754697; duration=0sec 2023-04-17 03:55:55,125 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:55,137 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/063c69030f424e6b8cdbdce8bb2e884e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/063c69030f424e6b8cdbdce8bb2e884e 2023-04-17 03:55:55,140 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 82824514d3f4f39e9f4f0e8595870ef8/C of 82824514d3f4f39e9f4f0e8595870ef8 into 063c69030f424e6b8cdbdce8bb2e884e(size=13.3 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:55:55,140 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:55,140 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8., storeName=82824514d3f4f39e9f4f0e8595870ef8/C, priority=13, startTime=1681703754697; duration=0sec 2023-04-17 03:55:55,140 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:55:55,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:55,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 82824514d3f4f39e9f4f0e8595870ef8, disabling compactions & flushes 2023-04-17 03:55:55,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. after waiting 0 ms 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:55,239 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 82824514d3f4f39e9f4f0e8595870ef8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=A 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=B 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 82824514d3f4f39e9f4f0e8595870ef8, store=C 2023-04-17 03:55:55,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:55,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:55,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6e62c1a032fd4e1b8ce27ee0bbf5c208 2023-04-17 03:55:55,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:56,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1b21287c817945c2a3b26371c626744c 2023-04-17 03:55:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:56,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=919 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/28385188ad474da2bee05a371ed5572d 2023-04-17 03:55:56,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/A/6e62c1a032fd4e1b8ce27ee0bbf5c208 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6e62c1a032fd4e1b8ce27ee0bbf5c208 2023-04-17 03:55:56,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6e62c1a032fd4e1b8ce27ee0bbf5c208, entries=100, sequenceid=919, filesize=9.5 K 2023-04-17 03:55:56,469 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/B/1b21287c817945c2a3b26371c626744c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1b21287c817945c2a3b26371c626744c 2023-04-17 03:55:56,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1b21287c817945c2a3b26371c626744c, entries=100, sequenceid=919, filesize=9.5 K 2023-04-17 03:55:56,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/.tmp/C/28385188ad474da2bee05a371ed5572d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/28385188ad474da2bee05a371ed5572d 2023-04-17 03:55:56,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/28385188ad474da2bee05a371ed5572d, entries=100, sequenceid=919, filesize=9.5 K 2023-04-17 03:55:56,476 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 82824514d3f4f39e9f4f0e8595870ef8 in 1237ms, sequenceid=919, compaction requested=true 2023-04-17 03:55:56,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9edd1a973da1461da8881be3786c3ab9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/33622698072b4d1498a52fb2e1efb597, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/8feb391460014957a2095a74a2415510, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a18fc6871b19445ca40e43fca604e820, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d2ddd013aa0d438e9b84cb2b5c9756f6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d99baeb997b04118ac98daacf8ad7d22, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/cb36a1802ecb47eea1c7735cb5d8f923, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7b13005edd944d10a70a6ccfbb94e44b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e889aeeb40fa44da831b80da62d7815a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a8816e10bd674abf8c0ed6730f7f7f3a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d] to archive 2023-04-17 03:55:56,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:56,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0a82896c95724310be5b70a68c378796 2023-04-17 03:55:56,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e60ece98c9df49e694a6fc070df1c098 2023-04-17 03:55:56,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6fe63d30edf2441e81f29ed2059ea971 2023-04-17 03:55:56,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/198d1f522c854f4e95f94148d79f6e01 2023-04-17 03:55:56,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1703f3467d94434896beda13a29c5745 2023-04-17 03:55:56,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9edd1a973da1461da8881be3786c3ab9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9edd1a973da1461da8881be3786c3ab9 2023-04-17 03:55:56,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/f181bf8507504204adafc977564cb169 2023-04-17 03:55:56,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/29a0020de3e74ea39e5ec60d7fc0892c 2023-04-17 03:55:56,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/33622698072b4d1498a52fb2e1efb597 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/33622698072b4d1498a52fb2e1efb597 2023-04-17 03:55:56,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/9555a52f36ef4203b653891e19442234 2023-04-17 03:55:56,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/fa919bbaf83648b088ebfa9a24269f86 2023-04-17 03:55:56,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/8feb391460014957a2095a74a2415510 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/8feb391460014957a2095a74a2415510 2023-04-17 03:55:56,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4c236b1df9a2407e945c8cc3d9e802df 2023-04-17 03:55:56,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5d554a0d11194235b37134ffad5bc0f2 2023-04-17 03:55:56,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a18fc6871b19445ca40e43fca604e820 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a18fc6871b19445ca40e43fca604e820 2023-04-17 03:55:56,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/3420f55316d1417d9dd3040cf5280849 2023-04-17 03:55:56,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cb700c4d8f04506bff4a3d1875cf88d 2023-04-17 03:55:56,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7aac72b0522c484fafdef7ed6e8f9baa 2023-04-17 03:55:56,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0955d6c4cad14339bbd850f759478761 2023-04-17 03:55:56,520 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d2ddd013aa0d438e9b84cb2b5c9756f6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d2ddd013aa0d438e9b84cb2b5c9756f6 2023-04-17 03:55:56,520 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/221f9f6dd63848178694d96cc7ba9f8f 2023-04-17 03:55:56,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/4499ff11e25243ef9986004d0918393c 2023-04-17 03:55:56,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d99baeb997b04118ac98daacf8ad7d22 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d99baeb997b04118ac98daacf8ad7d22 2023-04-17 03:55:56,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/227bf9c0fa03412f859601b621c49c50 2023-04-17 03:55:56,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5eba8bf27a0b4d66a0af8239e6b6e906 2023-04-17 03:55:56,524 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/44f83870249e495e9e6b4e7450f920c4 2023-04-17 03:55:56,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/cb36a1802ecb47eea1c7735cb5d8f923 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/cb36a1802ecb47eea1c7735cb5d8f923 2023-04-17 03:55:56,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a005058f20544c8c941851763fdc679f 2023-04-17 03:55:56,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/5baac366e1f5479cbcccfe2669a0b408 2023-04-17 03:55:56,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/d7a249f8c11541e9bf4d93b0af30df06 2023-04-17 03:55:56,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/749a0ef79c834ec695cfceda3f8d3089 2023-04-17 03:55:56,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7b13005edd944d10a70a6ccfbb94e44b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/7b13005edd944d10a70a6ccfbb94e44b 2023-04-17 03:55:56,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6468a92f72fb413e8fa2e9da545c70c5 2023-04-17 03:55:56,530 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/2dd7c7b3fef44234bf7124e86f719533 2023-04-17 03:55:56,531 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e889aeeb40fa44da831b80da62d7815a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e889aeeb40fa44da831b80da62d7815a 2023-04-17 03:55:56,532 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/df3318dda3cc4c09a86a3536913f2623 2023-04-17 03:55:56,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/1cadc5f60fdf473a8e669f3589d4aa7f 2023-04-17 03:55:56,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/42b1268e598d42a289f29c84b6f33526 2023-04-17 03:55:56,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a8816e10bd674abf8c0ed6730f7f7f3a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/a8816e10bd674abf8c0ed6730f7f7f3a 2023-04-17 03:55:56,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/53a6596aba3048988a4c8d3aca18c067 2023-04-17 03:55:56,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/87199aaeaf4c46548808a51971e0735d 2023-04-17 03:55:56,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2b2b83dd80c140b5ae8a5be6c1e28044, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2ce69741a0004fa582e6f1da68f0304e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0d27f6dce5164e5e9986c9bb55659f61, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c63b4556dbf24f368d016fceb005ec7a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e866cb3f1f2942f98081b03e9c64b19a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1dbab3773c4049f6b39c84d5991781dd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d2e5c868fc784687a2f1140d3d38efed, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/4d41eaebf9e34d96bc8023027aaab3b0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d75ffc95b9ef427db1db3f0a31cffeb3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e3444968122448008122023221efbf84, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6] to archive 2023-04-17 03:55:56,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:56,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/8a6756dc905a4a10a4cf84d70ab0cd57 2023-04-17 03:55:56,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89416847c27d420891050ed0a4b8ce67 2023-04-17 03:55:56,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9691e893b95a48818de67f53902ea7d1 2023-04-17 03:55:56,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f647007c7bf8421a99b09476318640ee 2023-04-17 03:55:56,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2b2b83dd80c140b5ae8a5be6c1e28044 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2b2b83dd80c140b5ae8a5be6c1e28044 2023-04-17 03:55:56,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/72b683b3c5004f4688840fa2cfbcbc95 2023-04-17 03:55:56,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/9ac48ca0ef60478a96f41bbccc44f74d 2023-04-17 03:55:56,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fdb77080873b40999d048f2c79004ad4 2023-04-17 03:55:56,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2ce69741a0004fa582e6f1da68f0304e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2ce69741a0004fa582e6f1da68f0304e 2023-04-17 03:55:56,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/02c316bbd6af468da56f8a6b062b231f 2023-04-17 03:55:56,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1af8e0b109e34dbda78421d1e6ea15c9 2023-04-17 03:55:56,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0d27f6dce5164e5e9986c9bb55659f61 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0d27f6dce5164e5e9986c9bb55659f61 2023-04-17 03:55:56,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/44db02f1140f4c4dbc1cec9bfbeeed0a 2023-04-17 03:55:56,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64954c355b4c445e80f8bf7022f3744a 2023-04-17 03:55:56,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c63b4556dbf24f368d016fceb005ec7a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c63b4556dbf24f368d016fceb005ec7a 2023-04-17 03:55:56,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/335d5092ef7b422497c4f6448860b027 2023-04-17 03:55:56,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/79fa31bbefbf4b2aa518d691d06169be 2023-04-17 03:55:56,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/89cecdeb56964a35819c0419fa8fed5e 2023-04-17 03:55:56,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e866cb3f1f2942f98081b03e9c64b19a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e866cb3f1f2942f98081b03e9c64b19a 2023-04-17 03:55:56,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/64cd7bc0b1cb4f6d953818d97bdf5a39 2023-04-17 03:55:56,559 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/3f8c2ad4e543416ebc369bf5ca0e2318 2023-04-17 03:55:56,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1dbab3773c4049f6b39c84d5991781dd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1dbab3773c4049f6b39c84d5991781dd 2023-04-17 03:55:56,560 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/158843ca54aa4bf5b96e0d1353f8824a 2023-04-17 03:55:56,561 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f204d58e1efb44888ea2661fdfa74000 2023-04-17 03:55:56,562 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/c3bb5b320e0247e0a2b4a5805493eea5 2023-04-17 03:55:56,563 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/583b351e7bde482f861a8cc6458e2f64 2023-04-17 03:55:56,564 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d2e5c868fc784687a2f1140d3d38efed to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d2e5c868fc784687a2f1140d3d38efed 2023-04-17 03:55:56,565 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/941803d55e614ae49fd4149dcaa2f25c 2023-04-17 03:55:56,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1c34c4c4b32a4727bfd062be76f02672 2023-04-17 03:55:56,566 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/0ee29a7622744cd49250cfafaf61e287 2023-04-17 03:55:56,567 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/4d41eaebf9e34d96bc8023027aaab3b0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/4d41eaebf9e34d96bc8023027aaab3b0 2023-04-17 03:55:56,568 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/977fab211c1f4c76a3ec718b9c8ffc77 2023-04-17 03:55:56,569 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/f80c31e999a2467a82c80ae5ef47fd16 2023-04-17 03:55:56,570 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d75ffc95b9ef427db1db3f0a31cffeb3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/d75ffc95b9ef427db1db3f0a31cffeb3 2023-04-17 03:55:56,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/963e6a03777141179b3106f8d5f83e13 2023-04-17 03:55:56,571 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/2d98d84f81794cf0bcc1f762fa418195 2023-04-17 03:55:56,572 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/377ef43dfde049bc93bf2b0769e072a9 2023-04-17 03:55:56,573 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e3444968122448008122023221efbf84 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/e3444968122448008122023221efbf84 2023-04-17 03:55:56,574 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1a86013d41394417aef6ccce590ddd6f 2023-04-17 03:55:56,575 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/195a5b6bfe66410d90623b33eb5e9942 2023-04-17 03:55:56,577 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/aabc6795ef7e49f79881a89d9365a2a6 2023-04-17 03:55:56,580 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/a60ffcc5e4834747b14e71ae5a2e1058, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bb60ebf894634c5cb7f5468c5b8ded05, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/00c96335eb0145e3a0c3c5716d8969ee, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0ca143830a849218756429dee880ee1, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/372a09ae991c4a4e88fec27fb966599d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0e5000d106042e8956a9edcfe972c28, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bf36b196bd79486682169098e8a5ef9e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/4d787af6cafc4d6a9795c5635ab5c6fb, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/1f01042632244e2a9e8076e0a30af510, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cbff4a1029ee47d4ad69b6c0d2ec8e26, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea] to archive 2023-04-17 03:55:56,581 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:55:56,582 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d16d9bbaf38b46cfb1e021b89937cef6 2023-04-17 03:55:56,583 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/95c2e45ff1764faa8e328af2c9851dbc 2023-04-17 03:55:56,584 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/8e4e126ab01d4af9b2c7148c96466f65 2023-04-17 03:55:56,585 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b3b41821c08b43908a37ba55f9f2c33e 2023-04-17 03:55:56,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/a60ffcc5e4834747b14e71ae5a2e1058 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/a60ffcc5e4834747b14e71ae5a2e1058 2023-04-17 03:55:56,586 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d015d53ed7a041e899021d15841030a6 2023-04-17 03:55:56,587 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6a13eda52aea48b8baff98e7f6c00dd1 2023-04-17 03:55:56,588 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/ea7d0eb85e3f4a36884f74b2453582d8 2023-04-17 03:55:56,589 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bb60ebf894634c5cb7f5468c5b8ded05 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bb60ebf894634c5cb7f5468c5b8ded05 2023-04-17 03:55:56,590 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d3f6243d145843b993e9125a804a60c2 2023-04-17 03:55:56,591 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c5ca6908f88e4bebb50c4fbd58dfc5b0 2023-04-17 03:55:56,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/00c96335eb0145e3a0c3c5716d8969ee to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/00c96335eb0145e3a0c3c5716d8969ee 2023-04-17 03:55:56,592 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/7655aceed82540b48d53fa04e2f0651a 2023-04-17 03:55:56,593 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/d8bf10c104564bac8ea5c8ec642e0da4 2023-04-17 03:55:56,594 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0ca143830a849218756429dee880ee1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0ca143830a849218756429dee880ee1 2023-04-17 03:55:56,595 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/208c69dfc95e43b9ab64675b11adfdbe 2023-04-17 03:55:56,596 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c96578188d99422ab8b550dbf1cb5d56 2023-04-17 03:55:56,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3851c3d2e318415bb030146c0ee0ef99 2023-04-17 03:55:56,597 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/372a09ae991c4a4e88fec27fb966599d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/372a09ae991c4a4e88fec27fb966599d 2023-04-17 03:55:56,598 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b42c0ddecef54bcca81f8c4d80c803aa 2023-04-17 03:55:56,599 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/197590dc87984074a1104b133038bedb 2023-04-17 03:55:56,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0e5000d106042e8956a9edcfe972c28 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/c0e5000d106042e8956a9edcfe972c28 2023-04-17 03:55:56,600 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/84312d5ee97f43e597fdf626bf2e0d2f 2023-04-17 03:55:56,601 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/6c3788703c6d4330ab1bce851521db35 2023-04-17 03:55:56,602 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/fe8817078a3f44588be29c7af5350220 2023-04-17 03:55:56,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cacdc51bc2034918a29d5ee600c7d649 2023-04-17 03:55:56,603 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bf36b196bd79486682169098e8a5ef9e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/bf36b196bd79486682169098e8a5ef9e 2023-04-17 03:55:56,604 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/5978865ad2824739bb21466b186dbc82 2023-04-17 03:55:56,605 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/556fb8265b234c7db1e6aec91b0d0705 2023-04-17 03:55:56,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/3a4191cd8cd24bd0ad3399d77172ba1b 2023-04-17 03:55:56,606 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/4d787af6cafc4d6a9795c5635ab5c6fb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/4d787af6cafc4d6a9795c5635ab5c6fb 2023-04-17 03:55:56,607 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b7b66a6c4d4445bf8fd987175baa67ec 2023-04-17 03:55:56,608 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/097da474899b4f9d9cd99a4d883c4c05 2023-04-17 03:55:56,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/1f01042632244e2a9e8076e0a30af510 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/1f01042632244e2a9e8076e0a30af510 2023-04-17 03:55:56,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dd1b8980cff84affb4b377efd50fe060 2023-04-17 03:55:56,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/2b55bebe5dd84395a97c1d83466be3b3 2023-04-17 03:55:56,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/513a39360204489e9553be20689c9113 2023-04-17 03:55:56,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cbff4a1029ee47d4ad69b6c0d2ec8e26 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/cbff4a1029ee47d4ad69b6c0d2ec8e26 2023-04-17 03:55:56,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/62ea5bb7c74742489d91b327151bcd44 2023-04-17 03:55:56,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/da661952077746989a7805271d5d7012 2023-04-17 03:55:56,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/dc604df01dbb41c89555cd6fcf3eb8ea 2023-04-17 03:55:56,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/recovered.edits/922.seqid, newMaxSeqId=922, maxSeqId=1 2023-04-17 03:55:56,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8. 2023-04-17 03:55:56,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 82824514d3f4f39e9f4f0e8595870ef8: 2023-04-17 03:55:56,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:56,621 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=82824514d3f4f39e9f4f0e8595870ef8, regionState=CLOSED 2023-04-17 03:55:56,621 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703756621"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703756621"}]},"ts":"1681703756621"} 2023-04-17 03:55:56,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-17 03:55:56,623 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 82824514d3f4f39e9f4f0e8595870ef8, server=jenkins-hbase4.apache.org,36381,1681703614253 in 1.5350 sec 2023-04-17 03:55:56,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-17 03:55:56,624 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=82824514d3f4f39e9f4f0e8595870ef8, UNASSIGN in 1.5390 sec 2023-04-17 03:55:56,624 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703756624"}]},"ts":"1681703756624"} 2023-04-17 03:55:56,625 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:55:56,626 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:55:56,627 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5510 sec 2023-04-17 03:55:57,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-17 03:55:57,185 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-17 03:55:57,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:55:57,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,187 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 03:55:57,188 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,193 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:57,195 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/recovered.edits] 2023-04-17 03:55:57,197 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0dfbfd75b0814c64b78ddfde9166aa0f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/0dfbfd75b0814c64b78ddfde9166aa0f 2023-04-17 03:55:57,198 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6e62c1a032fd4e1b8ce27ee0bbf5c208 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/6e62c1a032fd4e1b8ce27ee0bbf5c208 2023-04-17 03:55:57,199 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e5f861ea01844bdcb44302ad4bf8db6f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/A/e5f861ea01844bdcb44302ad4bf8db6f 2023-04-17 03:55:57,200 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1b21287c817945c2a3b26371c626744c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/1b21287c817945c2a3b26371c626744c 2023-04-17 03:55:57,201 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/ab5716418a9c4aa7b34e326d731ebcde to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/ab5716418a9c4aa7b34e326d731ebcde 2023-04-17 03:55:57,202 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fa79732383c347a0bdfc9f20b272d3cc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/B/fa79732383c347a0bdfc9f20b272d3cc 2023-04-17 03:55:57,203 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/063c69030f424e6b8cdbdce8bb2e884e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/063c69030f424e6b8cdbdce8bb2e884e 2023-04-17 03:55:57,204 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/28385188ad474da2bee05a371ed5572d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/28385188ad474da2bee05a371ed5572d 2023-04-17 03:55:57,205 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b0903c3206cc41ea98005005c5eb5e30 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/C/b0903c3206cc41ea98005005c5eb5e30 2023-04-17 03:55:57,207 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/recovered.edits/922.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8/recovered.edits/922.seqid 2023-04-17 03:55:57,208 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/82824514d3f4f39e9f4f0e8595870ef8 2023-04-17 03:55:57,208 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:57,210 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,211 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:55:57,212 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:55:57,213 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,213 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:55:57,213 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703757213"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:57,214 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:55:57,214 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 82824514d3f4f39e9f4f0e8595870ef8, NAME => 'TestAcidGuarantees,,1681703732530.82824514d3f4f39e9f4f0e8595870ef8.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:55:57,214 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:55:57,215 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703757214"}]},"ts":"9223372036854775807"} 2023-04-17 03:55:57,215 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:55:57,217 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-17 03:55:57,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-17 03:55:57,289 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-17 03:55:57,300 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=235 (was 227) - Thread LEAK? -, OpenFileDescriptor=583 (was 553) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=391 (was 354) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=986 (was 573) - AvailableMemoryMB LEAK? - 2023-04-17 03:55:57,312 INFO [Listener at localhost/44831] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=235, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=391, ProcessCount=169, AvailableMemoryMB=986 2023-04-17 03:55:57,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:55:57,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:55:57,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:57,317 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-17 03:55:57,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-17 03:55:57,318 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-17 03:55:57,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 03:55:57,319 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,319 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b empty. 2023-04-17 03:55:57,320 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,320 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:55:57,329 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:55:57,330 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 41c317bf95e4bf8c59626c6f1adb8a3b, NAME => 'TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp 2023-04-17 03:55:57,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:57,337 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 41c317bf95e4bf8c59626c6f1adb8a3b, disabling compactions & flushes 2023-04-17 03:55:57,338 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,338 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,338 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. after waiting 0 ms 2023-04-17 03:55:57,338 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,338 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,338 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:55:57,341 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-17 03:55:57,342 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703757342"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703757342"}]},"ts":"1681703757342"} 2023-04-17 03:55:57,343 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-17 03:55:57,343 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-17 03:55:57,343 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703757343"}]},"ts":"1681703757343"} 2023-04-17 03:55:57,344 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-17 03:55:57,348 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, ASSIGN}] 2023-04-17 03:55:57,350 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, ASSIGN 2023-04-17 03:55:57,351 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=false 2023-04-17 03:55:57,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 03:55:57,502 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:57,502 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703757502"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703757502"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703757502"}]},"ts":"1681703757502"} 2023-04-17 03:55:57,503 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:57,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 03:55:57,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 41c317bf95e4bf8c59626c6f1adb8a3b, NAME => 'TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:55:57,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:57,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,659 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,660 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A 2023-04-17 03:55:57,660 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A 2023-04-17 03:55:57,660 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:57,660 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName A 2023-04-17 03:55:57,661 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:57,661 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,662 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B 2023-04-17 03:55:57,662 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B 2023-04-17 03:55:57,662 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:57,662 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName B 2023-04-17 03:55:57,663 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:57,663 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,664 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C 2023-04-17 03:55:57,664 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C 2023-04-17 03:55:57,664 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:57,664 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName C 2023-04-17 03:55:57,664 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:57,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,667 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:55:57,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:57,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-17 03:55:57,670 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 41c317bf95e4bf8c59626c6f1adb8a3b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58941359, jitterRate=-0.1217053085565567}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:55:57,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:55:57,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., pid=56, masterSystemTime=1681703757655 2023-04-17 03:55:57,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,672 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:57,672 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:57,672 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703757672"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703757672"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703757672"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703757672"}]},"ts":"1681703757672"} 2023-04-17 03:55:57,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-17 03:55:57,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 170 msec 2023-04-17 03:55:57,675 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-17 03:55:57,675 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, ASSIGN in 326 msec 2023-04-17 03:55:57,676 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-17 03:55:57,676 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703757676"}]},"ts":"1681703757676"} 2023-04-17 03:55:57,677 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-17 03:55:57,679 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-17 03:55:57,680 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 365 msec 2023-04-17 03:55:57,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-17 03:55:57,921 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-17 03:55:57,922 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:57,927 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:57,928 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:57,929 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59826, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:57,930 DEBUG [Listener at localhost/44831] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-17 03:55:57,933 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58496, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-17 03:55:57,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-17 03:55:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-17 03:55:57,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-17 03:55:58,344 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:55:58,345 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-17 03:55:58,345 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-17 03:55:58,347 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-17 03:55:58,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, REOPEN/MOVE}] 2023-04-17 03:55:58,350 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, REOPEN/MOVE 2023-04-17 03:55:58,350 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,350 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703758350"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703758350"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703758350"}]},"ts":"1681703758350"} 2023-04-17 03:55:58,351 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:58,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,504 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 41c317bf95e4bf8c59626c6f1adb8a3b, disabling compactions & flushes 2023-04-17 03:55:58,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. after waiting 0 ms 2023-04-17 03:55:58,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-17 03:55:58,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:55:58,508 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 41c317bf95e4bf8c59626c6f1adb8a3b to self. 2023-04-17 03:55:58,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,510 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=CLOSED 2023-04-17 03:55:58,510 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703758509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703758509"}]},"ts":"1681703758509"} 2023-04-17 03:55:58,512 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-17 03:55:58,512 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 160 msec 2023-04-17 03:55:58,512 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36381,1681703614253; forceNewPlan=false, retain=true 2023-04-17 03:55:58,662 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,663 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703758662"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703758662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703758662"}]},"ts":"1681703758662"} 2023-04-17 03:55:58,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:55:58,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 41c317bf95e4bf8c59626c6f1adb8a3b, NAME => 'TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.', STARTKEY => '', ENDKEY => ''} 2023-04-17 03:55:58,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-17 03:55:58,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,820 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,821 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A 2023-04-17 03:55:58,821 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A 2023-04-17 03:55:58,821 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:58,821 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName A 2023-04-17 03:55:58,822 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:58,823 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,823 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B 2023-04-17 03:55:58,823 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B 2023-04-17 03:55:58,824 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:58,824 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName B 2023-04-17 03:55:58,824 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:58,824 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,825 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C 2023-04-17 03:55:58,825 DEBUG [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C 2023-04-17 03:55:58,825 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-17 03:55:58,825 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 41c317bf95e4bf8c59626c6f1adb8a3b columnFamilyName C 2023-04-17 03:55:58,826 INFO [StoreOpener-41c317bf95e4bf8c59626c6f1adb8a3b-1] regionserver.HStore(310): Store=41c317bf95e4bf8c59626c6f1adb8a3b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-17 03:55:58,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-17 03:55:58,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 41c317bf95e4bf8c59626c6f1adb8a3b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68467612, jitterRate=0.020246922969818115}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-17 03:55:58,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:55:58,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., pid=61, masterSystemTime=1681703758815 2023-04-17 03:55:58,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,832 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,832 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703758832"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681703758832"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681703758832"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681703758832"}]},"ts":"1681703758832"} 2023-04-17 03:55:58,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-17 03:55:58,834 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 169 msec 2023-04-17 03:55:58,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-17 03:55:58,835 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, REOPEN/MOVE in 485 msec 2023-04-17 03:55:58,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-17 03:55:58,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-17 03:55:58,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-17 03:55:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-17 03:55:58,840 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,844 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,845 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,849 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,849 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,852 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,853 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,862 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,862 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,866 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,867 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,873 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,873 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,879 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,879 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,886 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,887 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x0eedd898 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,891 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@308b3f39, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,891 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(139): Connect 0x618c0022 to 127.0.0.1:60620 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-17 03:55:58,895 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1453d93, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-17 03:55:58,904 DEBUG [hconnection-0xb2160ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,910 DEBUG [hconnection-0x7220ec55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,910 DEBUG [hconnection-0x13a7ccbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:58,911 DEBUG [hconnection-0x4e0bfdb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,911 DEBUG [hconnection-0x25b6662-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,911 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59858, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,911 DEBUG [hconnection-0x2418c744-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,911 DEBUG [hconnection-0x73216b4a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,911 DEBUG [hconnection-0x2c6402f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,914 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,917 DEBUG [hconnection-0x30fdd3a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,918 DEBUG [hconnection-0x537e0eac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59856, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59868, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59880, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59896, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59908, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59898, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,921 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-17 03:55:58,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:58,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:58,936 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:58,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:58,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:58,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:58,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:58,939 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:58,939 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:58,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:58,942 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:55:58,942 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:58,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:55:58,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:58,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:55:58,943 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:58,943 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:55:58,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:55:58,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:58,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:58,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:58,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:58,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:58,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:58,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:58,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:58,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c7a4b3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:58,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:58,953 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,955 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:58,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:58,957 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:58,957 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:58,961 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:58,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:58,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:58,961 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:58,961 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:58,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:58,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,963 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:58,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:58,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:58,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703758959, max timestamp=1681703758964Num uniques 859; 2023-04-17 03:55:58,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703758959, max timestamp=1681703758964Num uniques 859; 2023-04-17 03:55:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:55:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703758959, max timestamp=1681703758964Num uniques 859; 2023-04-17 03:55:58,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,967 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:58,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:58,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:58,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:58,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:58,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:58,970 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:58,971 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:58,971 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:58,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:58,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:58,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:58,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:58,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703818982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703818983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703818983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703818984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:58,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703818993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:58,995 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179ba1d6cb45804d389100476d80fef360 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179ba1d6cb45804d389100476d80fef360 2023-04-17 03:55:59,071 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:59,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:59,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:59,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:59,075 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:59,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:59,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:59,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:59,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:59,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:59,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:59,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:59,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:59,079 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:59,079 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:59,081 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:59,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5442fce5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:59,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:59,083 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,084 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:59,085 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:59,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,085 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:59,087 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:59,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703819088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,090 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:59,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:59,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:59,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:59,090 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:59,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:59,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703819088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703819088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,092 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,093 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703819092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:59,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,097 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,098 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:59,098 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:59,097 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,098 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703819095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,198 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:59,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:59,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:59,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:59,201 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:59,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:59,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:59,202 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,202 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,203 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,203 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:59,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:59,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:59,205 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,205 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:59,205 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:59,205 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:59,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,208 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,208 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:59,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448ba6fe[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:59,210 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:59,210 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,211 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:59,211 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:59,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:59,212 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:59,213 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,213 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:59,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:59,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:59,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:59,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:59,216 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:59,216 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:59,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,217 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,219 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:59,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,223 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:59,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:59,223 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,223 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:59,223 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703819291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703819293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703819293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703819295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703819300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,323 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:59,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:59,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:59,327 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:59,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:59,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:59,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,329 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,329 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:59,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:59,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:59,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:59,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:59,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:59,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:59,331 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:59,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,333 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:59,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2af2fb15[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:59,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:59,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:59,340 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:59,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:59,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:59,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:59,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:59,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:59,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:59,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:59,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:59,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:59,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:59,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,351 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:59,351 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:59,351 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:55:59,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,352 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,424 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/62d4451c4e064cb28354bd4f93607181 2023-04-17 03:55:59,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/4040b428968549349ea78801c213b593 2023-04-17 03:55:59,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/e67bf34fcec44852897731942849d269 2023-04-17 03:55:59,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/62d4451c4e064cb28354bd4f93607181 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181 2023-04-17 03:55:59,451 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:59,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:59,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:59,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181, entries=150, sequenceid=15, filesize=25.1 K 2023-04-17 03:55:59,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:59,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/4040b428968549349ea78801c213b593 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593 2023-04-17 03:55:59,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:59,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,457 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:59,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:59,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:59,460 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,460 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:59,460 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:59,460 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:55:59,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 03:55:59,460 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:55:59,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/e67bf34fcec44852897731942849d269 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269 2023-04-17 03:55:59,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,462 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269, entries=150, sequenceid=15, filesize=11.6 K 2023-04-17 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,468 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:55:59,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=138.65 KB/141980 for 41c317bf95e4bf8c59626c6f1adb8a3b in 527ms, sequenceid=15, compaction requested=false 2023-04-17 03:55:59,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:55:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f1530c4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:55:59,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:55:59,470 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,472 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:55:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:55:59,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,472 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:55:59,474 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:55:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:55:59,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:55:59,477 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:55:59,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:55:59,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:55:59,477 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:55:59,477 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:55:59,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,478 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:55:59,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:55:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:55:59,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:55:59,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:55:59,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:55:59,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,483 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:55:59,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:55:59,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:55:59,484 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:55:59,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:55:59,484 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,584 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:55:59,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:55:59,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:55:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:55:59,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:55:59,588 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:55:59,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:55:59,588 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:55:59,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,589 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:55:59,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:55:59,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,591 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:55:59,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:55:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:55:59,591 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:55:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:55:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:55:59,592 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:55:59,592 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:59,592 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:55:59,593 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:55:59,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:55:59,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:55:59,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703819599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703819602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703819602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703819602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703819603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,615 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304176c0e95d9a37c47ad8ca38d599387f4cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176c0e95d9a37c47ad8ca38d599387f4cb 2023-04-17 03:55:59,619 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/c262573fd6d44153bdc1ff2bd8d49625 2023-04-17 03:55:59,633 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/d6c3346b9c4c4a149fc6a780c27bb440 2023-04-17 03:55:59,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703819704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703819707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703819707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703819708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703819908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703819908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703819910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:55:59,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:55:59,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703819910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,041 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/a95102f303794c9381771b2c1a447aef 2023-04-17 03:56:00,044 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/c262573fd6d44153bdc1ff2bd8d49625 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625 2023-04-17 03:56:00,048 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625, entries=150, sequenceid=52, filesize=25.1 K 2023-04-17 03:56:00,049 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/d6c3346b9c4c4a149fc6a780c27bb440 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440 2023-04-17 03:56:00,052 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440, entries=150, sequenceid=52, filesize=11.6 K 2023-04-17 03:56:00,053 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/a95102f303794c9381771b2c1a447aef as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef 2023-04-17 03:56:00,056 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef, entries=150, sequenceid=52, filesize=11.6 K 2023-04-17 03:56:00,057 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=40.25 KB/41220 for 41c317bf95e4bf8c59626c6f1adb8a3b in 465ms, sequenceid=52, compaction requested=false 2023-04-17 03:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:00,057 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:00,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:00,060 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,060 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:00,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c735417[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:00,062 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:00,062 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,063 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:00,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:00,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:00,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,063 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:00,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:00,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:00,067 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:00,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:00,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:00,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:00,067 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:00,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,069 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,071 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,076 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,076 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:00,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:00,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:00,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,076 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:00,076 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:00,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:00,113 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:56:00,113 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:00,113 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,114 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:00,114 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,114 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:00,114 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,127 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177493d6eab34141d1b3b6587a5e9de1e1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177493d6eab34141d1b3b6587a5e9de1e1 2023-04-17 03:56:00,138 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/583081cea72c48eb8ac96da916e072aa 2023-04-17 03:56:00,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/52fdb5c8f28346d8b26b708459908b6f 2023-04-17 03:56:00,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/fc7511c18e07465fa0b8d76188056253 2023-04-17 03:56:00,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/583081cea72c48eb8ac96da916e072aa as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa 2023-04-17 03:56:00,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa, entries=150, sequenceid=62, filesize=25.1 K 2023-04-17 03:56:00,170 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/52fdb5c8f28346d8b26b708459908b6f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f 2023-04-17 03:56:00,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 03:56:00,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/fc7511c18e07465fa0b8d76188056253 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253 2023-04-17 03:56:00,176 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:00,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:00,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253, entries=150, sequenceid=62, filesize=11.6 K 2023-04-17 03:56:00,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 65ms, sequenceid=62, compaction requested=true 2023-04-17 03:56:00,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:00,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:00,178 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:00,178 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:00,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:00,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:00,179 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:00,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:00,179 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:00,179 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,179 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=75.4 K 2023-04-17 03:56:00,179 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:00,179 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:00,179 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,179 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 62d4451c4e064cb28354bd4f93607181, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681703758932 2023-04-17 03:56:00,180 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=34.8 K 2023-04-17 03:56:00,180 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c262573fd6d44153bdc1ff2bd8d49625, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703758977 2023-04-17 03:56:00,180 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4040b428968549349ea78801c213b593, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681703758932 2023-04-17 03:56:00,180 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 583081cea72c48eb8ac96da916e072aa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:00,180 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d6c3346b9c4c4a149fc6a780c27bb440, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703758977 2023-04-17 03:56:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:00,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:00,180 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:00,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:00,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:00,181 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 52fdb5c8f28346d8b26b708459908b6f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:00,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,181 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:00,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:00,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,182 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:00,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:00,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:00,184 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:00,189 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:00,190 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:00,190 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,190 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:00,190 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:00,191 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:00,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:00,192 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,193 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d6b331[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:00,194 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:00,194 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,195 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:00,196 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:00,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,196 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:00,198 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:00,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,198 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:00,201 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:00,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:00,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:00,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:00,201 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:00,205 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,205 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,205 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,207 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,210 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:00,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,210 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:00,210 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,210 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703820213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703820213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703820215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,218 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703820216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703820216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,221 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#512 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:00,224 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041730881ed77e6a4969bca215aef770b132 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041730881ed77e6a4969bca215aef770b132 2023-04-17 03:56:00,231 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/081bd0adc84a4e7aa6e0d4d0f03952fd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/081bd0adc84a4e7aa6e0d4d0f03952fd 2023-04-17 03:56:00,235 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into 081bd0adc84a4e7aa6e0d4d0f03952fd(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:00,235 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:00,235 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=13, startTime=1681703760178; duration=0sec 2023-04-17 03:56:00,235 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:00,235 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:00,236 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:00,236 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:00,236 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,236 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=34.8 K 2023-04-17 03:56:00,237 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting e67bf34fcec44852897731942849d269, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681703758932 2023-04-17 03:56:00,237 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting a95102f303794c9381771b2c1a447aef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1681703758977 2023-04-17 03:56:00,237 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fc7511c18e07465fa0b8d76188056253, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:00,241 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/8092e1f44a604e948d2f9317cbc33a9f 2023-04-17 03:56:00,242 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#513 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:00,310 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:00,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:00,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:00,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:00,314 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:00,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:00,314 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:00,315 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,315 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,316 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,316 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,316 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,316 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:00,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:00,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,318 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:00,318 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703820317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,318 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:00,318 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:00,319 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,319 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:00,319 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:00,319 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:00,319 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:00,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703820319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,322 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703820322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703820322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48c1c4cd[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:00,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:00,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:00,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:00,326 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:00,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:00,331 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:00,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:00,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:00,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:00,331 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:00,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,332 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,337 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:00,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:00,337 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,338 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,338 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,338 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:00,338 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:00,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:00,438 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:00,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:00,441 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:00,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:00,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:00,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,442 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,443 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,443 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:00,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:00,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:00,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:00,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:00,446 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:00,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:00,448 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,448 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:00,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e544e01[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:00,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:00,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,451 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:00,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,451 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:00,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:00,453 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,453 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,454 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:00,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:00,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:00,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:00,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:00,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:00,456 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:00,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,458 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,458 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,460 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,462 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:00,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:00,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:00,463 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,462 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,463 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,463 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,463 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,463 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:00,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703820520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703820523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703820525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703820526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,564 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:00,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:00,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:00,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:00,567 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:00,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:00,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:00,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,569 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:00,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:00,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:00,571 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,571 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:00,571 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:00,571 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:00,571 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,574 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,574 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:00,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38717830[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:00,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:00,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,577 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,577 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:00,578 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:00,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,578 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:00,579 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,579 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:00,581 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:00,581 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:00,581 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:00,581 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:00,581 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:00,583 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,583 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:00,583 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,584 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:00,585 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:00,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,590 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:00,591 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:00,591 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:00,591 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,590 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:00,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:00,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:00,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:00,658 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/ce08ade35d474a82a83ca5afc287f717 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/ce08ade35d474a82a83ca5afc287f717 2023-04-17 03:56:00,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/2f421d9ae1d94f04976be799e17feef2 2023-04-17 03:56:00,665 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into ce08ade35d474a82a83ca5afc287f717(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:00,666 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:00,666 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=13, startTime=1681703760178; duration=0sec 2023-04-17 03:56:00,666 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:00,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/eeebe55a09fc482b9e4a356a1c4fb42f 2023-04-17 03:56:00,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/8092e1f44a604e948d2f9317cbc33a9f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f 2023-04-17 03:56:00,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f, entries=150, sequenceid=82, filesize=25.1 K 2023-04-17 03:56:00,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/2f421d9ae1d94f04976be799e17feef2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2 2023-04-17 03:56:00,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2, entries=150, sequenceid=82, filesize=11.6 K 2023-04-17 03:56:00,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/eeebe55a09fc482b9e4a356a1c4fb42f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f 2023-04-17 03:56:00,691 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:00,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:00,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f, entries=150, sequenceid=82, filesize=11.6 K 2023-04-17 03:56:00,693 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 41c317bf95e4bf8c59626c6f1adb8a3b in 511ms, sequenceid=82, compaction requested=false 2023-04-17 03:56:00,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:00,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:00,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:00,695 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,696 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:00,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:00,696 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,697 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,697 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:00,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:00,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:00,698 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:00,698 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:00,698 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:00,698 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:00,699 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:00,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:00,721 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:00,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703820766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703820823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703820826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703820828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703820830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:00,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:00,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703820869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703821072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,119 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417e94b8c45e7454134b1d2935cc326fd3e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e94b8c45e7454134b1d2935cc326fd3e 2023-04-17 03:56:01,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703821327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703821331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703821332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703821334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703821376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,454 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/226d11695e8d4df5b1a2011b4a7fba7b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/226d11695e8d4df5b1a2011b4a7fba7b 2023-04-17 03:56:01,458 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into 226d11695e8d4df5b1a2011b4a7fba7b(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-17 03:56:01,458 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:01,458 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=13, startTime=1681703760178; duration=1sec 2023-04-17 03:56:01,458 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:01,525 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/9b7f5cdc374747e9bac595c84ccc0dc6 2023-04-17 03:56:01,534 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/2d2aa1339b744b57a1525f607fad8dcd 2023-04-17 03:56:01,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:01,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703821877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,943 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/db83c4f67c464bef986f25f45b3fd7f0 2023-04-17 03:56:01,947 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/9b7f5cdc374747e9bac595c84ccc0dc6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6 2023-04-17 03:56:01,950 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6, entries=150, sequenceid=97, filesize=25.1 K 2023-04-17 03:56:01,951 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/2d2aa1339b744b57a1525f607fad8dcd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd 2023-04-17 03:56:01,954 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd, entries=150, sequenceid=97, filesize=11.6 K 2023-04-17 03:56:01,955 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/db83c4f67c464bef986f25f45b3fd7f0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0 2023-04-17 03:56:01,958 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0, entries=150, sequenceid=97, filesize=11.6 K 2023-04-17 03:56:01,959 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1261ms, sequenceid=97, compaction requested=true 2023-04-17 03:56:01,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:01,959 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:01,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:01,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:01,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:01,959 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:01,961 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:01,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,961 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:01,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:01,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fe1b34[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:01,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:01,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,964 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:01,964 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:01,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:01,966 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:01,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:01,969 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:01,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:01,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:01,969 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:01,969 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:01,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,970 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:01,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,971 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:01,971 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,973 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:01,975 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:01,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:01,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:01,975 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:01,975 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:01,976 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:01,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,076 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,079 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,082 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,082 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,083 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,083 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,084 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:02,084 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,098 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a6ab9a9fd71a4cd2870ed12b820e341d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a6ab9a9fd71a4cd2870ed12b820e341d 2023-04-17 03:56:02,105 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/96f7d80ed15f45e68884a77f8fac65df 2023-04-17 03:56:02,117 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/41d56d1295864c9cb7b3d472a98b89d2 2023-04-17 03:56:02,126 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b5c22bdfd6724317b18a01c01ab6a01f 2023-04-17 03:56:02,131 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/96f7d80ed15f45e68884a77f8fac65df as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df 2023-04-17 03:56:02,135 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df, entries=150, sequenceid=117, filesize=25.1 K 2023-04-17 03:56:02,136 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/41d56d1295864c9cb7b3d472a98b89d2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2 2023-04-17 03:56:02,139 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 03:56:02,139 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b5c22bdfd6724317b18a01c01ab6a01f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f 2023-04-17 03:56:02,144 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f, entries=150, sequenceid=117, filesize=11.6 K 2023-04-17 03:56:02,145 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 41c317bf95e4bf8c59626c6f1adb8a3b in 61ms, sequenceid=117, compaction requested=true 2023-04-17 03:56:02,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,145 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,147 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e5f5f4a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,149 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,149 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,150 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,151 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,152 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-17 03:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,160 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,160 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,163 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,163 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,163 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,167 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,167 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,167 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,170 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,172 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,172 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,173 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,173 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,273 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,277 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,277 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,278 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,278 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,281 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,281 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,281 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,281 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,281 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,283 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52973ac6[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,284 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,285 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,287 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,287 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,287 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,287 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,287 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,288 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,289 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,289 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,292 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,292 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,292 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,294 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,295 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,296 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,303 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,302 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,303 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,303 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:02,361 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:02,362 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:02,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703822388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703822389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703822390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703822390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,399 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177b385b9f35a7482cb7e2781ab36e9d5d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b385b9f35a7482cb7e2781ab36e9d5d 2023-04-17 03:56:02,403 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,409 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,409 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,410 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/b24f58f84cbc42b6bacc2eed74e0362b 2023-04-17 03:56:02,411 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,411 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,413 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:02,416 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,416 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,424 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,425 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,426 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,426 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6114685c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,428 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,429 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,430 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,430 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,430 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,431 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,431 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,434 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,434 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,435 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,436 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,437 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,437 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,439 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,440 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,440 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,440 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,440 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/ef6216ef5b7d4927a594320150464141 2023-04-17 03:56:02,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/8e073b8c617645eb84b978eff33de1c2 2023-04-17 03:56:02,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/b24f58f84cbc42b6bacc2eed74e0362b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b 2023-04-17 03:56:02,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b, entries=450, sequenceid=130, filesize=65.9 K 2023-04-17 03:56:02,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/ef6216ef5b7d4927a594320150464141 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141 2023-04-17 03:56:02,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141, entries=150, sequenceid=130, filesize=11.7 K 2023-04-17 03:56:02,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/8e073b8c617645eb84b978eff33de1c2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2 2023-04-17 03:56:02,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2, entries=150, sequenceid=130, filesize=11.7 K 2023-04-17 03:56:02,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 41c317bf95e4bf8c59626c6f1adb8a3b in 121ms, sequenceid=130, compaction requested=true 2023-04-17 03:56:02,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:02,481 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:56:02,481 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:56:02,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:02,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:02,482 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 170581 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:56:02,482 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:02,483 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,483 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/226d11695e8d4df5b1a2011b4a7fba7b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=166.6 K 2023-04-17 03:56:02,483 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:56:02,483 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:02,483 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,483 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/081bd0adc84a4e7aa6e0d4d0f03952fd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=58.2 K 2023-04-17 03:56:02,484 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 226d11695e8d4df5b1a2011b4a7fba7b, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:02,484 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 081bd0adc84a4e7aa6e0d4d0f03952fd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:02,484 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 8092e1f44a604e948d2f9317cbc33a9f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681703760157 2023-04-17 03:56:02,485 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2f421d9ae1d94f04976be799e17feef2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681703760157 2023-04-17 03:56:02,485 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9b7f5cdc374747e9bac595c84ccc0dc6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681703760195 2023-04-17 03:56:02,485 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 2d2aa1339b744b57a1525f607fad8dcd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681703760195 2023-04-17 03:56:02,485 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 96f7d80ed15f45e68884a77f8fac65df, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703760756 2023-04-17 03:56:02,486 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 41d56d1295864c9cb7b3d472a98b89d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703760756 2023-04-17 03:56:02,486 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting b24f58f84cbc42b6bacc2eed74e0362b, keycount=450, bloomtype=ROW, size=65.9 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:02,486 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ef6216ef5b7d4927a594320150464141, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:02,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:02,499 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 03:56:02,499 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762367, max timestamp=1681703762497Num uniques 859; 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762367, max timestamp=1681703762497Num uniques 859; 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762367, max timestamp=1681703762497Num uniques 859; 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:02,500 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,500 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:02,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:02,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,503 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#525 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:02,509 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/da0ae208b6e54359928386cd1b3a3284 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/da0ae208b6e54359928386cd1b3a3284 2023-04-17 03:56:02,518 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into da0ae208b6e54359928386cd1b3a3284(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:02,518 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,518 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=11, startTime=1681703762481; duration=0sec 2023-04-17 03:56:02,518 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:02,518 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-17 03:56:02,519 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59633 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-17 03:56:02,519 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:02,519 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,519 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/ce08ade35d474a82a83ca5afc287f717, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=58.2 K 2023-04-17 03:56:02,520 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting ce08ade35d474a82a83ca5afc287f717, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681703759599 2023-04-17 03:56:02,520 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting eeebe55a09fc482b9e4a356a1c4fb42f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681703760157 2023-04-17 03:56:02,520 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting db83c4f67c464bef986f25f45b3fd7f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1681703760195 2023-04-17 03:56:02,521 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b5c22bdfd6724317b18a01c01ab6a01f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1681703760756 2023-04-17 03:56:02,534 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8e073b8c617645eb84b978eff33de1c2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:02,539 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304175ff2139e97e14649bf62d895508f617e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175ff2139e97e14649bf62d895508f617e 2023-04-17 03:56:02,540 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762534, max timestamp=1681703762541Num uniques 859; 2023-04-17 03:56:02,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762534, max timestamp=1681703762541Num uniques 859; 2023-04-17 03:56:02,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703762534, max timestamp=1681703762541Num uniques 859; 2023-04-17 03:56:02,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:02,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,545 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,545 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,546 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#528 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:02,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,547 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,547 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,548 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,548 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,558 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=152, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a0c3358b095d4fd193060adbec78c80d 2023-04-17 03:56:02,559 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/0894866ac00a4d0abd2d10b69a6646a4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/0894866ac00a4d0abd2d10b69a6646a4 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:02,560 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,560 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,563 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,563 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69bbe8f1[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,566 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,566 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,566 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 0894866ac00a4d0abd2d10b69a6646a4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:02,566 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,566 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=11, startTime=1681703762481; duration=0sec 2023-04-17 03:56:02,567 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:02,567 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,567 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,568 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,569 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703762561, max timestamp=1681703762567Num uniques 1018; 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703762561, max timestamp=1681703762567Num uniques 1018; 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681703762561, max timestamp=1681703762567Num uniques 1018; 2023-04-17 03:56:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-17 03:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,572 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,572 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,572 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/3c072ab0b9124cdf8a4ec409d056cc94 2023-04-17 03:56:02,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,575 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,576 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,577 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,580 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,580 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,581 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,581 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/76de685ca446497f901b1d2a827083f7 2023-04-17 03:56:02,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a0c3358b095d4fd193060adbec78c80d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d 2023-04-17 03:56:02,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d, entries=150, sequenceid=152, filesize=25.3 K 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703762582, max timestamp=1681703762590Num uniques 1027; 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703762582, max timestamp=1681703762590Num uniques 1027; 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681703762582, max timestamp=1681703762590Num uniques 1027; 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:56:02,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-17 03:56:02,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/3c072ab0b9124cdf8a4ec409d056cc94 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94 2023-04-17 03:56:02,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94, entries=150, sequenceid=152, filesize=11.8 K 2023-04-17 03:56:02,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/76de685ca446497f901b1d2a827083f7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7 2023-04-17 03:56:02,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7, entries=150, sequenceid=152, filesize=11.8 K 2023-04-17 03:56:02,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 114ms, sequenceid=152, compaction requested=false 2023-04-17 03:56:02,613 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:02,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=120.76 KB heapSize=334.08 KB 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:02,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:02,629 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417982d20db856045ac8709f5b840de85e5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417982d20db856045ac8709f5b840de85e5 2023-04-17 03:56:02,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703822652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703822653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703822653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703822653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,681 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,684 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,684 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,685 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,685 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,686 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,686 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,686 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,686 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,687 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:02,689 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,691 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,691 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dda4f83[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,693 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,694 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,695 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,696 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,699 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,699 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,699 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,699 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,699 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,701 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,703 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,705 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,705 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,705 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,705 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703822759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703822759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703822759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703822759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,806 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,809 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,809 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,812 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,812 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,812 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,812 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,813 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,813 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:02,814 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,814 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,818 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,818 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46f011f4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,819 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,821 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,821 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,821 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,821 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,823 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,826 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,826 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,826 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,829 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,832 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,834 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,836 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,837 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,837 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,837 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,837 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,837 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703822886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,937 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:02,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:02,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:02,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:02,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:02,941 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:02,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:02,942 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,942 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,944 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,945 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:02,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:02,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:02,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:02,945 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:02,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,945 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:02,946 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:02,946 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:02,946 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:02,946 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/ac033cfd048e4bfda444acdd8de821dc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac033cfd048e4bfda444acdd8de821dc 2023-04-17 03:56:02,947 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:02,948 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,951 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into ac033cfd048e4bfda444acdd8de821dc(size=25.5 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:02,951 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:02,951 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=11, startTime=1681703762481; duration=0sec 2023-04-17 03:56:02,951 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e4d7fe[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:02,951 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:02,952 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,953 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,954 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:02,954 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:02,955 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,955 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:02,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:02,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:02,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:02,958 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:02,958 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:02,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:02,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,959 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:02,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,960 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,962 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:02,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703822962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:02,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:02,965 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:02,965 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:02,965 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703822963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,965 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703822963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:02,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703822964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:02,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:02,966 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:02,968 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,033 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/6e3fe7432b954c55bda8bd40ffa562af 2023-04-17 03:56:03,065 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:03,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:03,069 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:03,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:03,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:03,069 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,072 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,072 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:03,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:03,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:03,074 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:03,075 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:03,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ae5bf3[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:03,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:03,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,080 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,080 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,081 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:03,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:03,083 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:03,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,083 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:03,087 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:03,087 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:03,087 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:03,087 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:03,087 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:03,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,089 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,090 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,091 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:03,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,094 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,095 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:03,095 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:03,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,094 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,095 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,195 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:03,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:03,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:03,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:03,198 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:03,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:03,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:03,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,200 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,200 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:03,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:03,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:03,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:03,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:03,202 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:03,203 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:03,204 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,204 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,205 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d77ac72[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:03,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:03,206 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,211 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:03,211 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,211 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:03,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:03,212 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,212 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:03,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:03,215 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:03,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:03,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:03,215 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:03,215 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:03,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,216 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,217 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,218 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,221 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:03,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,222 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,221 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,222 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:03,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,222 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:03,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703823267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703823267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703823269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703823269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,322 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:03,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:03,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:03,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:03,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:03,325 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:03,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:03,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:03,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,327 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:03,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:03,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:03,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:03,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:03,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,329 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:03,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:03,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:03,330 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:03,330 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:03,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,331 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,332 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:03,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4e6b99[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:03,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:03,333 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,334 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:03,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:03,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:03,334 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:03,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:03,341 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,341 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:03,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:03,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:03,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:03,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:03,344 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:03,344 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:03,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,345 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,346 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,346 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,347 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:03,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:03,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,350 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:03,350 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:03,350 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,350 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,351 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/7e535735b793403f8d6517ddd1e43cfd 2023-04-17 03:56:03,450 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:03,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:03,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:03,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:03,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:03,464 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:03,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/03303405fbcc41968cb90171b43d19fd 2023-04-17 03:56:03,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,465 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,466 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,466 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,467 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,467 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:03,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:03,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/6e3fe7432b954c55bda8bd40ffa562af as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af 2023-04-17 03:56:03,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:03,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af, entries=150, sequenceid=221, filesize=25.3 K 2023-04-17 03:56:03,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/7e535735b793403f8d6517ddd1e43cfd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd 2023-04-17 03:56:03,476 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:03,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b71629[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:03,478 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:03,478 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,479 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:03,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,479 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:03,481 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,481 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:03,483 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:03,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:03,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:03,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:03,483 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:03,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd, entries=150, sequenceid=221, filesize=11.8 K 2023-04-17 03:56:03,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/03303405fbcc41968cb90171b43d19fd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd 2023-04-17 03:56:03,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:03,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,487 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:03,491 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd, entries=150, sequenceid=221, filesize=11.8 K 2023-04-17 03:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~334.03 KB/342048, currentSize=67.09 KB/68700 for 41c317bf95e4bf8c59626c6f1adb8a3b in 876ms, sequenceid=221, compaction requested=true 2023-04-17 03:56:03,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:03,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:03,492 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:03,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:03,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:03,492 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:03,493 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77912 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:03,493 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:03,493 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:03,493 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:03,493 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,493 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,493 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac033cfd048e4bfda444acdd8de821dc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=76.1 K 2023-04-17 03:56:03,493 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/da0ae208b6e54359928386cd1b3a3284, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=35.5 K 2023-04-17 03:56:03,494 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ac033cfd048e4bfda444acdd8de821dc, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:03,494 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting da0ae208b6e54359928386cd1b3a3284, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:03,494 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a0c3358b095d4fd193060adbec78c80d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681703762389 2023-04-17 03:56:03,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,494 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,495 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 6e3fe7432b954c55bda8bd40ffa562af, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:03,495 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 3c072ab0b9124cdf8a4ec409d056cc94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681703762389 2023-04-17 03:56:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:03,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:03,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:03,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:03,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,495 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:03,495 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:03,495 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:03,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:03,496 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7e535735b793403f8d6517ddd1e43cfd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:03,501 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:03,502 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#534 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:03,595 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:03,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:03,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:03,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:03,598 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:03,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:03,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:03,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,599 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,601 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:03,601 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:03,601 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,601 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:03,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:03,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:03,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:03,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:03,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:03,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:03,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:03,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:03,603 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:03,603 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:03,603 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:03,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:03,604 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:03,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:03,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:03,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703823792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703823792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703823793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703823793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703823896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703823896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703823897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:03,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703823897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:03,926 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/deee300b34644fe8aa482523b68b9618 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/deee300b34644fe8aa482523b68b9618 2023-04-17 03:56:03,932 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into deee300b34644fe8aa482523b68b9618(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:03,932 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:03,932 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=13, startTime=1681703763492; duration=0sec 2023-04-17 03:56:03,932 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:03,932 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:03,933 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:03,934 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:03,934 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:03,934 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/0894866ac00a4d0abd2d10b69a6646a4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=35.5 K 2023-04-17 03:56:03,934 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 0894866ac00a4d0abd2d10b69a6646a4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1681703762347 2023-04-17 03:56:03,937 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 76de685ca446497f901b1d2a827083f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1681703762389 2023-04-17 03:56:03,938 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 03303405fbcc41968cb90171b43d19fd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:03,943 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#537 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:04,035 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417154548a4d4b2430ea656dad53087095e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417154548a4d4b2430ea656dad53087095e 2023-04-17 03:56:04,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703824098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703824098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703824101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703824101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,349 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/c2b2cfd515b24ff2a0b78356550fc9bc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c2b2cfd515b24ff2a0b78356550fc9bc 2023-04-17 03:56:04,352 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/53bcb6b2b0b347ca992e4ae785519ae2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/53bcb6b2b0b347ca992e4ae785519ae2 2023-04-17 03:56:04,354 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into c2b2cfd515b24ff2a0b78356550fc9bc(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:04,354 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:04,354 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=13, startTime=1681703763492; duration=0sec 2023-04-17 03:56:04,354 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:04,357 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 53bcb6b2b0b347ca992e4ae785519ae2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:04,357 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:04,357 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=13, startTime=1681703763492; duration=0sec 2023-04-17 03:56:04,357 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:04,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703824402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703824402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703824404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703824406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,441 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=235, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/f78f755cb7ef4cbe9a760697130afc0c 2023-04-17 03:56:04,450 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/97c1455569c24b42bc127dd3dcf5317a 2023-04-17 03:56:04,460 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/73adb9fd77f4433b96ebb9210e672949 2023-04-17 03:56:04,464 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/f78f755cb7ef4cbe9a760697130afc0c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c 2023-04-17 03:56:04,467 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c, entries=150, sequenceid=235, filesize=25.3 K 2023-04-17 03:56:04,468 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/97c1455569c24b42bc127dd3dcf5317a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a 2023-04-17 03:56:04,472 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a, entries=150, sequenceid=235, filesize=11.8 K 2023-04-17 03:56:04,472 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/73adb9fd77f4433b96ebb9210e672949 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949 2023-04-17 03:56:04,475 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949, entries=150, sequenceid=235, filesize=11.8 K 2023-04-17 03:56:04,476 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 41c317bf95e4bf8c59626c6f1adb8a3b in 873ms, sequenceid=235, compaction requested=false 2023-04-17 03:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:04,476 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,476 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:04,477 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:04,478 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,479 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,479 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,479 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:04,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:04,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6986b12f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:04,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:04,481 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,483 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:04,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:04,483 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:04,483 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,484 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:04,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:04,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:04,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:04,486 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:04,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:04,488 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:04,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:04,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:04,488 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:04,488 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:04,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,492 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:04,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:04,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:04,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:04,493 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,501 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,501 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:04,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:04,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:04,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:04,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:04,501 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:04,501 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:04,502 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:04,502 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:04,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:04,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:04,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:04,502 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,602 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:04,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:04,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:04,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:04,606 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:04,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:04,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:04,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:04,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,606 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:04,607 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,607 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:04,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:04,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,608 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:04,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:04,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:04,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:04,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:04,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:04,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:04,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:04,609 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:04,610 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:04,610 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:04,610 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:04,624 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041740083152266e4e17b6f94ea88a4adcbd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740083152266e4e17b6f94ea88a4adcbd 2023-04-17 03:56:04,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:04,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:04,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703824913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703824914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703824915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:04,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703824916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:04,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703824915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,030 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/286c626ca085473aa1c800363ca16e61 2023-04-17 03:56:05,040 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/047d1db55a0a4985b072903482dec526 2023-04-17 03:56:05,049 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b28d13a5cf034ce2864642202052cd18 2023-04-17 03:56:05,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/286c626ca085473aa1c800363ca16e61 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61 2023-04-17 03:56:05,055 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61, entries=150, sequenceid=258, filesize=25.3 K 2023-04-17 03:56:05,056 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/047d1db55a0a4985b072903482dec526 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526 2023-04-17 03:56:05,058 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526, entries=150, sequenceid=258, filesize=11.8 K 2023-04-17 03:56:05,059 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b28d13a5cf034ce2864642202052cd18 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18 2023-04-17 03:56:05,062 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18, entries=150, sequenceid=258, filesize=11.8 K 2023-04-17 03:56:05,063 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 41c317bf95e4bf8c59626c6f1adb8a3b in 453ms, sequenceid=258, compaction requested=true 2023-04-17 03:56:05,063 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:05,063 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:05,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:05,064 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,065 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,065 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:05,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19628451[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:05,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:05,067 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:05,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:05,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:05,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,068 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:05,070 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:05,070 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:05,070 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,070 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:05,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:05,073 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:05,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:05,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:05,073 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:05,073 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:05,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:05,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,075 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,076 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,079 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:05,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,079 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:05,079 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:05,079 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:05,080 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:05,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:05,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:05,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:05,180 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:05,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:05,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:05,183 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:05,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:05,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:05,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:05,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,185 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:05,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:05,185 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:05,185 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,185 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:05,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:05,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:05,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:05,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:05,186 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:05,187 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:05,187 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,201 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171902e9f9b24d4a0a85fa73127f0bb5f3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171902e9f9b24d4a0a85fa73127f0bb5f3 2023-04-17 03:56:05,215 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/462dd707593c4510a2e87ca3203f4ac0 2023-04-17 03:56:05,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:05,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,630 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/dfd332dd069643a5a8fbb20a56ced617 2023-04-17 03:56:05,639 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/593ebe12a5cc441abf41ac61a5748b6e 2023-04-17 03:56:05,643 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/462dd707593c4510a2e87ca3203f4ac0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0 2023-04-17 03:56:05,646 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0, entries=150, sequenceid=271, filesize=25.4 K 2023-04-17 03:56:05,647 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/dfd332dd069643a5a8fbb20a56ced617 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617 2023-04-17 03:56:05,651 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 03:56:05,652 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/593ebe12a5cc441abf41ac61a5748b6e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e 2023-04-17 03:56:05,655 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e, entries=150, sequenceid=271, filesize=11.9 K 2023-04-17 03:56:05,655 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 468ms, sequenceid=271, compaction requested=true 2023-04-17 03:56:05,656 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:05,656 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:05,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:05,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:05,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:05,656 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:05,658 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,658 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,659 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:05,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61083647[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:05,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:05,661 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,662 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:05,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:05,662 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:05,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,662 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:05,664 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,664 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:05,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:05,667 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:05,667 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:05,667 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:05,667 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:05,667 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:05,669 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,669 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:05,669 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:05,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:05,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:05,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:05,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:05,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:05,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,675 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:05,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:05,674 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:05,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:05,675 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,675 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:05,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:05,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:05,675 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:05,775 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:05,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:05,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:05,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:05,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:05,779 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:05,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:05,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:05,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:05,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,779 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,780 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:05,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:05,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:05,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:05,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:05,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:05,783 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:05,783 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:05,800 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041766517ccbe3fe4c54b78820d82dd3b9fb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041766517ccbe3fe4c54b78820d82dd3b9fb 2023-04-17 03:56:05,806 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/0d10a7c8bef140b89831273c8ef3f2a9 2023-04-17 03:56:05,817 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/8e8837f677d74ef89433337fd05876a2 2023-04-17 03:56:05,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:05,847 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:05,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703825959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703825960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703825960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703825961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:05,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:05,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703825962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703826163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703826163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703826164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59896 deadline: 1681703826164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703826166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,226 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/d19f1b7ee6164defbd34d49944e5f326 2023-04-17 03:56:06,229 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/0d10a7c8bef140b89831273c8ef3f2a9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9 2023-04-17 03:56:06,233 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9, entries=150, sequenceid=290, filesize=25.4 K 2023-04-17 03:56:06,233 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/8e8837f677d74ef89433337fd05876a2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2 2023-04-17 03:56:06,236 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2, entries=150, sequenceid=290, filesize=11.9 K 2023-04-17 03:56:06,237 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/d19f1b7ee6164defbd34d49944e5f326 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326 2023-04-17 03:56:06,240 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326, entries=150, sequenceid=290, filesize=11.9 K 2023-04-17 03:56:06,241 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 41c317bf95e4bf8c59626c6f1adb8a3b in 457ms, sequenceid=290, compaction requested=true 2023-04-17 03:56:06,241 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:06,241 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:06,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:06,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:06,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:06,241 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:06,242 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:06,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:06,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:06,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:06,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:06,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@701f6211[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:06,244 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,245 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:06,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:06,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:06,245 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,246 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:06,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:06,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:06,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:06,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:06,251 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:06,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:06,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:06,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:06,251 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:06,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,252 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:06,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:06,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:06,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,253 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:06,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:06,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,254 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:06,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:06,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:06,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:06,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:06,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:06,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:06,260 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:06,260 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:06,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:06,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:06,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:06,360 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:06,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:06,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:06,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:06,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:06,363 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:06,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:06,363 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:06,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:06,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,364 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:06,366 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:06,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:06,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,366 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:06,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:06,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:06,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:06,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:06,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:06,367 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:06,368 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:06,368 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:06,368 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:06,369 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:06,384 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304177b5e3ed3bc094b758a5c3d71b6848efc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b5e3ed3bc094b758a5c3d71b6848efc 2023-04-17 03:56:06,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:06,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:06,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703826483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703826483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703826487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703826488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703826489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703826590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703826590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703826593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703826595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703826595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,790 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=303, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a4fb9392f3104d2aab7beab8020b719d 2023-04-17 03:56:06,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703826794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703826795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703826797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703826799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:06,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:06,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703826799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703827097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703827101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703827101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703827102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703827107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,206 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/681f2c38f3724f668ae77f76d1662765 2023-04-17 03:56:07,233 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/8dd5394f3ade4a01877d43c10edd2735 2023-04-17 03:56:07,237 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a4fb9392f3104d2aab7beab8020b719d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d 2023-04-17 03:56:07,241 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d, entries=150, sequenceid=303, filesize=25.4 K 2023-04-17 03:56:07,241 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/681f2c38f3724f668ae77f76d1662765 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765 2023-04-17 03:56:07,249 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 03:56:07,250 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/8dd5394f3ade4a01877d43c10edd2735 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735 2023-04-17 03:56:07,253 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735, entries=150, sequenceid=303, filesize=11.9 K 2023-04-17 03:56:07,254 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 886ms, sequenceid=303, compaction requested=true 2023-04-17 03:56:07,254 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:07,254 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:07,256 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:07,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:07,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:07,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cccaaf4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:07,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:07,258 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,259 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:07,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:07,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:07,263 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:07,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:07,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:07,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:07,263 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:07,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,266 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:07,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:07,267 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,269 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:07,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:07,271 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:07,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:07,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:07,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:07,271 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:07,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:07,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,272 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:07,272 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:07,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:07,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:07,373 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:07,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:07,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:07,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:07,377 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:07,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:07,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:07,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:07,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:07,379 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:07,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:07,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:07,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:07,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:07,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:07,381 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:07,381 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:07,381 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:07,396 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041794e5f79636394e0792441d88f344949e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041794e5f79636394e0792441d88f344949e 2023-04-17 03:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:07,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:07,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703827612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703827613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703827615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703827615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703827615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703827717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703827717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703827720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703827720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703827720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,804 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=322, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/36cedddd05154932b36d251d3612eac9 2023-04-17 03:56:07,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703827919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703827920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703827925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703827925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:07,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:07,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703827925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,215 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/53353788463c4c29b97825a5413855ad 2023-04-17 03:56:08,226 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1ee5188f6b6d41bfa35f3f13b07a5016 2023-04-17 03:56:08,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59868 deadline: 1681703828225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59880 deadline: 1681703828226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,230 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/36cedddd05154932b36d251d3612eac9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9 2023-04-17 03:56:08,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59916 deadline: 1681703828231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703828231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703828231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,234 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9, entries=150, sequenceid=322, filesize=25.4 K 2023-04-17 03:56:08,234 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/53353788463c4c29b97825a5413855ad as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad 2023-04-17 03:56:08,238 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad, entries=150, sequenceid=322, filesize=11.9 K 2023-04-17 03:56:08,239 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1ee5188f6b6d41bfa35f3f13b07a5016 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016 2023-04-17 03:56:08,241 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016, entries=150, sequenceid=322, filesize=11.9 K 2023-04-17 03:56:08,242 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 41c317bf95e4bf8c59626c6f1adb8a3b in 861ms, sequenceid=322, compaction requested=true 2023-04-17 03:56:08,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:08,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:08,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:08,242 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:08,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:08,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:08,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:08,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:08,249 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:08,249 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,249 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:08,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:08,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2181f330[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:08,250 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:08,251 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,253 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,254 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,254 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:08,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:08,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:08,254 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:08,255 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,255 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:08,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:08,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:08,256 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:08,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:08,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:08,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:08,259 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:08,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:08,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:08,259 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:08,259 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:08,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:08,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,261 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:08,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:08,263 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:08,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:08,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:08,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:08,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:08,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:08,268 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:08,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:08,268 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:08,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:08,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:08,368 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:08,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:08,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:08,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:08,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:08,379 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:08,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:08,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:08,379 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:08,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,380 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:08,381 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:08,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:08,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:08,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,381 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:08,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:08,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:08,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:08,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:08,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:08,382 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:08,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:08,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:08,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:08,383 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:08,383 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:08,383 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:08,384 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:08,384 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:08,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:08,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:08,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703828750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703828750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703828753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703828754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703828754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,801 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417ded7d173d4624823ae027c07b85f185b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ded7d173d4624823ae027c07b85f185b 2023-04-17 03:56:08,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703828855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703828855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703828857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703828857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:08,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703828859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703829058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703829058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703829062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703829062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703829063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,206 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/47fa6fc411a24689b6dbc46a4f6cd1e2 2023-04-17 03:56:09,219 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/c8b29de07b4141418aabebe62d716c8e 2023-04-17 03:56:09,227 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/4d7055279dd14ee49594140f8b96792a 2023-04-17 03:56:09,230 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/47fa6fc411a24689b6dbc46a4f6cd1e2 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2 2023-04-17 03:56:09,235 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2, entries=150, sequenceid=335, filesize=25.4 K 2023-04-17 03:56:09,235 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/c8b29de07b4141418aabebe62d716c8e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e 2023-04-17 03:56:09,238 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e, entries=150, sequenceid=335, filesize=11.9 K 2023-04-17 03:56:09,239 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/4d7055279dd14ee49594140f8b96792a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a 2023-04-17 03:56:09,242 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a, entries=150, sequenceid=335, filesize=11.9 K 2023-04-17 03:56:09,242 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 41c317bf95e4bf8c59626c6f1adb8a3b in 859ms, sequenceid=335, compaction requested=true 2023-04-17 03:56:09,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:09,242 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,243 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,244 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,244 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,245 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11b5349[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,246 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,246 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,248 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,248 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,250 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,250 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,253 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,253 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,253 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,253 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,253 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,254 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,255 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,255 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,255 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,256 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,260 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,261 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,261 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,261 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,261 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,260 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,261 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:09,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,361 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:09,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:09,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:09,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:09,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:09,365 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,365 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:09,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:09,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:09,368 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,368 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,369 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:09,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:09,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:09,371 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,371 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:09,371 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:09,371 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,371 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,372 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,373 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,373 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9fa691[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,376 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,376 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,377 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,377 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,377 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,378 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,378 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,379 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703829375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703829375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703829376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,384 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,385 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,385 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703829381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,387 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,387 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,387 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,387 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304179e7cdcbe98d241d69df32daf23e2c51d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179e7cdcbe98d241d69df32daf23e2c51d 2023-04-17 03:56:09,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,388 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703829382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,393 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,393 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,394 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,393 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,394 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703829483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703829483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703829483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703829488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703829489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,494 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:09,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:09,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:09,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:09,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:09,497 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:09,497 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:09,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:09,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,498 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,500 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,500 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,501 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:09,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:09,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:09,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:09,502 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:09,502 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,502 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:09,502 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:09,503 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,503 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,503 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,503 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,503 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,503 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,504 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43cd363c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,506 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,510 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,510 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,510 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,512 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,515 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,515 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,515 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,517 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,518 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,520 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,521 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:09,521 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,521 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,621 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:09,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:09,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:09,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:09,625 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:09,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:09,625 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:09,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,626 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,628 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,628 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:09,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:09,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:09,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:09,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:09,629 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:09,630 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,630 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:09,630 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:09,630 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,632 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d7b4506[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,634 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,636 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,637 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,637 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,638 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,641 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,641 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,644 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,644 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,644 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,649 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,649 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,649 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,649 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703829689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703829689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703829690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703829692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703829692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,749 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:09,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:09,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:09,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:09,753 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:09,753 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,754 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:09,754 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:09,754 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,754 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,755 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,755 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,756 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:09,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:09,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:09,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:09,759 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,769 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,770 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a1d2fb[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,772 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,772 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,773 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,773 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,773 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,775 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,775 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,778 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,778 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,778 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,779 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,780 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,781 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,784 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,785 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,785 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,785 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,785 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,793 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/cbbac9c10dcf48e188657e8a88d161e8 2023-04-17 03:56:09,885 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:09,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:09,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:09,888 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:09,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:09,888 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:09,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,890 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,890 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,890 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,890 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:09,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:09,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:09,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:09,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:09,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:09,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:09,892 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,892 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:09,892 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:09,892 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:09,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:09,893 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41fd5190[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:09,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:09,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,896 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:09,896 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:09,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,897 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:09,898 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:09,899 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,899 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:09,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:09,901 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:09,901 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:09,901 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:09,901 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:09,901 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:09,902 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,902 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,902 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:09,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,903 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,904 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:09,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:09,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:09,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:09,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:09,906 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:09,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:09,907 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:09,906 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:09,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:09,907 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:09,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:09,907 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:09,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703829994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703829994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703829996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:09,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:09,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703829996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703829998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,008 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,011 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,011 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,013 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,013 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,013 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,013 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,014 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:10,015 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:10,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:10,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:10,015 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:10,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,018 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:10,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b2be345[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:10,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:10,019 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,020 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:10,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:10,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,021 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:10,021 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:10,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:10,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,022 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:10,022 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:10,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:10,025 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:10,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:10,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:10,025 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:10,025 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:10,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,026 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,027 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,029 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,031 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,032 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,032 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:10,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:10,033 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,033 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:10,033 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:10,133 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,137 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,137 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,138 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,139 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,139 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,140 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:10,141 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:10,141 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:10,143 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,143 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,144 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:10,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccfc853[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:10,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:10,145 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,150 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,150 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:10,150 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:10,150 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:10,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,150 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:10,152 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:10,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,152 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:10,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:10,154 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:10,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:10,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:10,154 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:10,154 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:10,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,157 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,157 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,161 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:10,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:10,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,162 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,162 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:10,162 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,162 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/48695568ecfa4d4c8332d5ac9701502f 2023-04-17 03:56:10,262 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,265 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,265 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,266 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,266 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,268 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,268 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,269 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:10,270 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:10,270 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:10,272 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:10,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:10,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c1ef3b[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:10,274 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:10,274 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,275 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:10,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,275 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:10,277 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:10,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,277 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:10,279 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:10,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:10,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:10,279 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:10,279 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:10,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,282 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,283 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,285 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,285 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,285 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,286 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:10,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,286 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,286 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:10,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:10,286 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,386 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,390 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,390 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,391 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,391 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,392 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,393 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,393 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,393 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:10,394 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:10,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:10,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:10,394 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:10,395 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,395 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:10,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f9b53e2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:10,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:10,397 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,399 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:10,400 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:10,400 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:10,401 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:10,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:10,403 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:10,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:10,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:10,404 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:10,404 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:10,405 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,405 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,405 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,406 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,407 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,410 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:10,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:10,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,410 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:10,410 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:10,410 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:10,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59892 deadline: 1681703830499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703830500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703830502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703830503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:10,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703830507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,510 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,513 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,513 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,514 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,516 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,516 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,517 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,517 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,518 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,518 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,518 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:10,518 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:10,518 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:10,520 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,520 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,521 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:10,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:10,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9eb09f[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:10,522 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:10,522 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,523 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,523 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:10,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,524 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:10,524 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:10,524 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:10,531 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,532 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:10,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:10,534 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:10,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:10,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:10,534 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:10,534 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:10,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,535 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,536 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:10,536 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:10,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:10,537 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:10,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:10,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:10,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:10,539 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,539 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:10,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,540 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:10,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,540 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,540 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:10,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:10,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:10,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:10,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1b31b705d4574eacb596ecef031b8167 2023-04-17 03:56:10,616 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/cbbac9c10dcf48e188657e8a88d161e8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8 2023-04-17 03:56:10,622 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8, entries=200, sequenceid=356, filesize=32.3 K 2023-04-17 03:56:10,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/48695568ecfa4d4c8332d5ac9701502f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f 2023-04-17 03:56:10,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f, entries=150, sequenceid=356, filesize=11.9 K 2023-04-17 03:56:10,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1b31b705d4574eacb596ecef031b8167 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167 2023-04-17 03:56:10,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167, entries=150, sequenceid=356, filesize=11.9 K 2023-04-17 03:56:10,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1263ms, sequenceid=356, compaction requested=true 2023-04-17 03:56:10,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:10,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:10,630 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-17 03:56:10,630 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-17 03:56:10,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:10,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:10,632 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 109600 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-17 03:56:10,632 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 241380 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-17 03:56:10,632 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:10,632 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:10,632 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,633 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,633 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/deee300b34644fe8aa482523b68b9618, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=107.0 K 2023-04-17 03:56:10,633 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c2b2cfd515b24ff2a0b78356550fc9bc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=235.7 K 2023-04-17 03:56:10,633 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting deee300b34644fe8aa482523b68b9618, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:10,633 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c2b2cfd515b24ff2a0b78356550fc9bc, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:10,633 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 97c1455569c24b42bc127dd3dcf5317a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681703762622 2023-04-17 03:56:10,633 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f78f755cb7ef4cbe9a760697130afc0c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681703762622 2023-04-17 03:56:10,634 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 047d1db55a0a4985b072903482dec526, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681703763788 2023-04-17 03:56:10,634 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 286c626ca085473aa1c800363ca16e61, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681703763788 2023-04-17 03:56:10,634 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting dfd332dd069643a5a8fbb20a56ced617, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681703764913 2023-04-17 03:56:10,634 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 462dd707593c4510a2e87ca3203f4ac0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681703764913 2023-04-17 03:56:10,634 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8e8837f677d74ef89433337fd05876a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681703765231 2023-04-17 03:56:10,635 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 0d10a7c8bef140b89831273c8ef3f2a9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681703765231 2023-04-17 03:56:10,635 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 681f2c38f3724f668ae77f76d1662765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703765854 2023-04-17 03:56:10,635 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 53353788463c4c29b97825a5413855ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681703766482 2023-04-17 03:56:10,635 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting a4fb9392f3104d2aab7beab8020b719d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703765854 2023-04-17 03:56:10,635 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c8b29de07b4141418aabebe62d716c8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681703767611 2023-04-17 03:56:10,636 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 36cedddd05154932b36d251d3612eac9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681703766482 2023-04-17 03:56:10,636 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 48695568ecfa4d4c8332d5ac9701502f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768752 2023-04-17 03:56:10,636 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 47fa6fc411a24689b6dbc46a4f6cd1e2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681703767611 2023-04-17 03:56:10,636 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting cbbac9c10dcf48e188657e8a88d161e8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768746 2023-04-17 03:56:10,640 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:10,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:10,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:10,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:10,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:10,643 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:10,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:10,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:10,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,644 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:10,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:10,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:10,646 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:10,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:10,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:10,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:10,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:10,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:10,648 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:10,648 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:10,648 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:10,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:10,649 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:10,651 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#561 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:10,672 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#562 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:11,072 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/b6b07c4710bc4c0c95f8d353f3d4ed31 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b6b07c4710bc4c0c95f8d353f3d4ed31 2023-04-17 03:56:11,076 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into b6b07c4710bc4c0c95f8d353f3d4ed31(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:11,076 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:11,076 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=7, startTime=1681703770630; duration=0sec 2023-04-17 03:56:11,076 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:11,076 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-17 03:56:11,078 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 109600 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-17 03:56:11,079 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:11,079 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:11,079 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/53bcb6b2b0b347ca992e4ae785519ae2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=107.0 K 2023-04-17 03:56:11,079 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 53bcb6b2b0b347ca992e4ae785519ae2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681703762604 2023-04-17 03:56:11,079 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 73adb9fd77f4433b96ebb9210e672949, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681703762622 2023-04-17 03:56:11,080 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b28d13a5cf034ce2864642202052cd18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1681703763788 2023-04-17 03:56:11,081 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 593ebe12a5cc441abf41ac61a5748b6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681703764913 2023-04-17 03:56:11,081 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d19f1b7ee6164defbd34d49944e5f326, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1681703765231 2023-04-17 03:56:11,081 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8dd5394f3ade4a01877d43c10edd2735, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1681703765854 2023-04-17 03:56:11,082 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1ee5188f6b6d41bfa35f3f13b07a5016, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1681703766482 2023-04-17 03:56:11,083 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417fb03f8b23a30458fa0861589f03a323e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417fb03f8b23a30458fa0861589f03a323e 2023-04-17 03:56:11,083 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4d7055279dd14ee49594140f8b96792a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681703767611 2023-04-17 03:56:11,084 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1b31b705d4574eacb596ecef031b8167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768752 2023-04-17 03:56:11,100 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#564 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:11,106 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/5358c89db7dc49328317a524f98b474c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5358c89db7dc49328317a524f98b474c 2023-04-17 03:56:11,109 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 5358c89db7dc49328317a524f98b474c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:11,110 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:11,110 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=7, startTime=1681703770630; duration=0sec 2023-04-17 03:56:11,110 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:11,490 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=367, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/4ed77c9bb77148c59250c3e4f383569e 2023-04-17 03:56:11,498 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/970e26657660495781755bdbffbd8dbd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/970e26657660495781755bdbffbd8dbd 2023-04-17 03:56:11,503 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into 970e26657660495781755bdbffbd8dbd(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:11,503 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:11,503 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=7, startTime=1681703770630; duration=0sec 2023-04-17 03:56:11,503 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:11,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:11,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:11,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703831527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703771521, max timestamp=1681703771528Num uniques 859; 2023-04-17 03:56:11,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:56:11,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703771521, max timestamp=1681703771528Num uniques 859; 2023-04-17 03:56:11,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:56:11,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703771521, max timestamp=1681703771528Num uniques 859; 2023-04-17 03:56:11,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:56:11,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703831541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703831542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703831542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703831543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703831631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703831646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703831646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703831646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703831647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703831835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703831849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703831849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703831850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:11,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703831850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,902 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/91e8c60f1b774aa3ab392713d8da3137 2023-04-17 03:56:11,911 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/c2b75fb7c79840ef9f3417ead97160ff 2023-04-17 03:56:11,915 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/4ed77c9bb77148c59250c3e4f383569e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e 2023-04-17 03:56:11,918 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e, entries=150, sequenceid=367, filesize=25.4 K 2023-04-17 03:56:11,919 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/91e8c60f1b774aa3ab392713d8da3137 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137 2023-04-17 03:56:11,922 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 03:56:11,923 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/c2b75fb7c79840ef9f3417ead97160ff as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff 2023-04-17 03:56:11,926 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff, entries=150, sequenceid=367, filesize=11.9 K 2023-04-17 03:56:11,926 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1278ms, sequenceid=367, compaction requested=false 2023-04-17 03:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:11,926 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:11,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:11,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:11,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:11,927 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:11,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:11,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:11,929 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:11,930 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:11,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:11,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59b172ef[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:11,931 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:11,931 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,932 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:11,932 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:11,932 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,932 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:11,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:11,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:11,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:11,936 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:11,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:11,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:11,936 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:11,936 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:11,937 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,937 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:11,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,938 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:11,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,939 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,942 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,942 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:11,942 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:11,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:11,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:11,942 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:11,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:11,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:11,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:11,943 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:11,943 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:11,943 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:11,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:11,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:11,942 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:11,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:11,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:12,043 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:12,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:12,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:12,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:12,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:12,048 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:12,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:12,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:12,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:12,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,048 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:12,050 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,050 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:12,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:12,050 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,050 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:12,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:12,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:12,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:12,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:12,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:12,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:12,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:12,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:12,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:12,051 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:12,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:12,051 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:12,051 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:12,052 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:12,072 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304170951d300914a4b0bbb9ad58edfddb429 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170951d300914a4b0bbb9ad58edfddb429 2023-04-17 03:56:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:12,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:12,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703832156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703832156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703832156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703832157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703832159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703832260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703832261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703832261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703832263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703832464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703832464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703832464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703832465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,477 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/aee7288306534561bec25f31a4a3a7f9 2023-04-17 03:56:12,501 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/17174ec04aaa4b0fa809efa505f477b8 2023-04-17 03:56:12,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703832661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703832767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703832768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703832769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:12,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703832769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,910 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/07c4284811d94130afa3286544f16dde 2023-04-17 03:56:12,914 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/aee7288306534561bec25f31a4a3a7f9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9 2023-04-17 03:56:12,917 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9, entries=150, sequenceid=404, filesize=25.4 K 2023-04-17 03:56:12,918 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/17174ec04aaa4b0fa809efa505f477b8 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8 2023-04-17 03:56:12,921 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8, entries=150, sequenceid=404, filesize=11.9 K 2023-04-17 03:56:12,922 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/07c4284811d94130afa3286544f16dde as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde 2023-04-17 03:56:12,925 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde, entries=150, sequenceid=404, filesize=11.9 K 2023-04-17 03:56:12,925 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 41c317bf95e4bf8c59626c6f1adb8a3b in 874ms, sequenceid=404, compaction requested=true 2023-04-17 03:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:12,925 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:12,926 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:12,927 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,927 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,928 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e2f6d5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:12,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:12,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,931 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:12,931 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,931 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:12,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:12,936 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:12,936 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:12,936 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:12,936 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:12,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:12,938 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:12,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:12,938 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:12,939 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:12,939 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:12,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:12,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:12,941 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,943 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,945 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:12,945 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:12,945 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:12,946 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:12,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:12,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:12,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:12,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:12,946 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:12,946 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:13,047 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:13,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:13,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:13,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:13,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:13,054 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:13,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:13,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:13,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:13,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:13,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:13,055 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:13,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:13,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,056 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:13,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:13,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:13,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:13,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:13,057 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:13,058 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:13,058 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:13,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:13,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703773323, max timestamp=1681703773353Num uniques 859; 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681703773323, max timestamp=1681703773353Num uniques 859; 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681703773323, max timestamp=1681703773353Num uniques 859; 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-17 03:56:13,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-17 03:56:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703833393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703833393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,397 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703833394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703833396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,481 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304171b4f217cc3a849d7b34295e3c1d47798 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171b4f217cc3a849d7b34295e3c1d47798 2023-04-17 03:56:13,485 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=415, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/2f100d9ee5ae4ffb9724f114a230ca35 2023-04-17 03:56:13,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703833498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703833498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703833499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703833499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703833668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703833700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703833701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703833701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:13,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703833705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:13,895 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/fc55427d7000419a8fccc87cd58a4ca9 2023-04-17 03:56:14,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703834003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703834005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703834005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703834007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,305 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/4532a05f04bc4cc2bb17ee5d8e3e2939 2023-04-17 03:56:14,310 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/2f100d9ee5ae4ffb9724f114a230ca35 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35 2023-04-17 03:56:14,313 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35, entries=150, sequenceid=415, filesize=25.4 K 2023-04-17 03:56:14,314 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/fc55427d7000419a8fccc87cd58a4ca9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9 2023-04-17 03:56:14,317 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 03:56:14,318 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/4532a05f04bc4cc2bb17ee5d8e3e2939 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939 2023-04-17 03:56:14,321 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939, entries=150, sequenceid=415, filesize=11.9 K 2023-04-17 03:56:14,321 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1263ms, sequenceid=415, compaction requested=true 2023-04-17 03:56:14,321 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:14,321 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:14,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:14,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:14,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:14,322 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,324 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:14,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:14,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f3a728[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:14,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:14,325 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,326 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:14,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:14,327 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:14,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,327 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:14,328 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:14,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:14,331 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:14,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:14,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:14,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:14,331 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:14,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:14,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,333 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:14,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:14,334 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,335 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:14,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:14,339 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:14,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:14,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:14,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,340 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:14,340 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:14,340 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:14,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:14,340 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:14,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:14,440 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:14,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:14,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:14,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:14,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:14,445 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:14,449 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:14,449 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:14,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:14,450 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:14,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:14,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:14,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:14,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:14,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:14,451 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:14,451 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:14,451 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:14,451 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=129.71 KB heapSize=365.45 KB 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:14,452 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:14,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:14,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:14,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703834514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703834515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703834516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703834519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703834620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703834620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703834620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703834623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703834822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703834823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703834823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:14,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703834828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:14,871 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041738ad1afd84f64732b4ddc2768f1ee88d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041738ad1afd84f64732b4ddc2768f1ee88d 2023-04-17 03:56:15,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703835127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703835128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703835128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703835135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,275 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=450, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/33c3b999e7364a59b14a91ae3215cfa6 2023-04-17 03:56:15,284 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/c0844a8c2bb54f1a825649f1e2ceda5e 2023-04-17 03:56:15,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703835631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703835631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703835632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703835640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703835684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,688 DEBUG [Thread-3796] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:56:15,696 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e 2023-04-17 03:56:15,700 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/33c3b999e7364a59b14a91ae3215cfa6 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6 2023-04-17 03:56:15,703 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6, entries=150, sequenceid=450, filesize=25.4 K 2023-04-17 03:56:15,704 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/c0844a8c2bb54f1a825649f1e2ceda5e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e 2023-04-17 03:56:15,707 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e, entries=150, sequenceid=450, filesize=11.9 K 2023-04-17 03:56:15,707 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e 2023-04-17 03:56:15,710 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e, entries=150, sequenceid=450, filesize=11.9 K 2023-04-17 03:56:15,711 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~365.41 KB/374176, currentSize=53.67 KB/54960 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1260ms, sequenceid=450, compaction requested=true 2023-04-17 03:56:15,711 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:15,711 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:15,714 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,715 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:15,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29e7e101[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:15,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:15,716 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,717 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:15,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:15,718 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:15,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,718 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:15,719 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:15,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:15,722 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:15,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:15,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:15,722 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:15,722 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:15,725 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,725 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,725 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,726 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,727 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:15,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:15,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,731 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:15,732 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:15,731 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,731 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:15,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:15,832 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:15,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:15,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:15,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:15,835 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:15,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:15,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:15,835 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:15,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,836 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,837 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:15,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:15,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:15,837 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,838 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:15,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:15,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:15,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:15,839 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:15,839 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:15,839 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:15,840 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:15,854 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417811d89df25c04a5c939369ec50ceae56 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417811d89df25c04a5c939369ec50ceae56 2023-04-17 03:56:15,869 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/19059b4cb73d491b8018a2c056bfed64 2023-04-17 03:56:15,886 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/1774af70db224839b22fd5345ea8a8ff 2023-04-17 03:56:15,895 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/5b44f8b2b2a44a35a210018519d381db 2023-04-17 03:56:15,899 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/19059b4cb73d491b8018a2c056bfed64 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64 2023-04-17 03:56:15,902 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64, entries=150, sequenceid=461, filesize=25.4 K 2023-04-17 03:56:15,903 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/1774af70db224839b22fd5345ea8a8ff as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff 2023-04-17 03:56:15,906 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff, entries=150, sequenceid=461, filesize=11.9 K 2023-04-17 03:56:15,907 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/5b44f8b2b2a44a35a210018519d381db as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db 2023-04-17 03:56:15,910 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db, entries=150, sequenceid=461, filesize=11.9 K 2023-04-17 03:56:15,911 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 41c317bf95e4bf8c59626c6f1adb8a3b in 71ms, sequenceid=461, compaction requested=true 2023-04-17 03:56:15,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:15,911 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:15,911 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,913 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,913 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b13d8d8[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:15,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:15,915 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,916 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:15,917 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:15,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:15,919 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,919 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:15,920 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:15,922 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:15,922 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:15,922 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:15,922 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:15,923 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:15,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:15,924 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:15,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:15,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:15,925 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,926 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:15,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:15,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:15,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:15,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:15,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:15,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:15,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:15,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:15,928 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:15,928 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,929 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:15,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:15,929 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,029 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,033 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,033 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,034 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,035 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,035 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,036 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:16,037 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,037 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,039 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,039 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,040 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ee2dd31[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,041 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,042 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,043 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,043 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,044 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,044 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,047 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,047 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,047 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,049 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,049 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,050 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,054 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,054 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,054 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,054 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,155 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,158 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,158 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,159 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,160 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,160 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,161 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,162 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,162 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,162 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:16,162 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,165 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6412eb22[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,167 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,168 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,168 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,170 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,170 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,172 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,173 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,173 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,173 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,173 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,174 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,174 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,174 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,175 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,176 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,178 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,178 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,179 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,179 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,279 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,289 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,289 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,291 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,291 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,292 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:16,293 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,293 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,296 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,296 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726bf148[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,299 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,299 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,300 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,301 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,301 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,301 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,301 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,302 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,302 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,305 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,305 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,305 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,307 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,307 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,307 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,309 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,312 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,313 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,312 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,313 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,313 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,313 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,313 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,413 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,416 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,416 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,417 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,418 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,418 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,419 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:16,420 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,420 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,421 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,422 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,422 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,423 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@175b32c5[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,424 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,424 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,425 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,426 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,426 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,428 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,431 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,431 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,432 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,432 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,432 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,433 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,434 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,437 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,438 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,438 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,438 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,438 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,438 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,538 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,541 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,541 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,542 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,546 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,546 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:16,548 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,548 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,550 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,550 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f987e4a[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,552 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,553 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,553 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,553 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,554 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,554 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,557 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,557 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,557 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,557 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,557 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,560 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,560 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,560 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,561 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,562 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,565 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,565 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,566 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,566 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,566 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,666 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,669 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,669 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,670 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:16,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-17 03:56:16,671 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,671 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,671 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:16,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:16,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,672 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:16,672 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:16,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,672 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,673 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,675 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,675 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,675 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,675 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,675 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:16,676 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,676 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,676 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,676 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,676 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,676 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,677 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca0c560[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,680 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,680 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,682 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,683 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,683 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,683 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,683 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,683 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,684 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,688 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,688 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,689 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,689 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,690 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,691 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304173a44df3fc9ef4e4f96cc4fe5e9dad637 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173a44df3fc9ef4e4f96cc4fe5e9dad637 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,691 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,692 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,694 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,695 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,695 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,695 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,695 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703776719, max timestamp=1681703776727Num uniques 859; 2023-04-17 03:56:16,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703836725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:56:16,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703776719, max timestamp=1681703776727Num uniques 859; 2023-04-17 03:56:16,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681703776719, max timestamp=1681703776727Num uniques 859; 2023-04-17 03:56:16,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:56:16,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-17 03:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703836756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703836757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703836758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,795 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,799 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,799 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,802 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,802 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,804 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,804 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,804 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:16,804 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,804 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,806 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,807 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,807 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,807 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c900d[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,808 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,808 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,809 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,810 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,811 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,814 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,814 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,814 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,818 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,818 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,818 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,819 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,820 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,822 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,822 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,823 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,823 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,823 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:16,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703836829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703836862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703836862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703836864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,923 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:16,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:16,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:16,927 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:16,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:16,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:16,927 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,928 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,929 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,929 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,930 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:16,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:16,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:16,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:16,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:16,931 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,931 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:16,931 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:16,931 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:16,931 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:16,933 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,933 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:16,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fef47ff[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:16,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:16,935 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,940 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,940 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:16,940 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:16,940 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:16,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,940 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:16,941 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:16,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:16,944 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:16,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:16,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:16,944 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:16,944 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:16,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:16,946 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:16,947 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,948 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:16,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:16,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:16,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:16,950 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:16,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:16,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:16,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:16,950 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:16,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:16,951 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:16,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:16,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:16,951 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:16,951 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703837033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,051 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:17,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:17,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:17,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:17,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:17,055 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:17,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:17,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:17,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,055 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,059 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,059 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,060 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,060 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:17,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:17,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:17,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:17,061 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,061 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:17,061 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:17,061 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,061 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:17,062 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:17,063 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,063 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,064 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:17,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53190a66[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:17,065 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:17,065 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,066 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:17,066 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,066 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:17,066 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:17,068 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,068 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:17,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703837066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703837067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703837067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:17,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:17,070 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:17,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:17,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:17,070 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:17,070 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:17,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,073 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,074 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,075 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,077 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:17,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:17,077 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,078 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,078 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:17,078 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:17,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:17,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,098 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=477, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/438b123cf60b4165b69c71a9059c5766 2023-04-17 03:56:17,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/d6c9f6d5af79498cb9768a7b39b266a7 2023-04-17 03:56:17,178 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:17,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:17,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:17,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:17,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:17,182 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:17,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:17,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:17,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,182 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,183 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,183 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,184 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:17,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:17,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:17,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:17,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:17,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:17,185 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:17,186 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:17,187 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,187 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79b85ea0[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:17,189 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:17,189 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,191 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:17,191 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,191 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,193 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,193 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:17,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:17,195 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:17,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:17,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:17,195 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:17,195 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:17,197 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,197 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,197 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,197 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,199 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,201 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,201 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,201 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:17,201 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:17,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:17,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,202 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,202 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,202 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:17,202 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,202 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:17,203 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,203 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,203 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,303 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:17,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:17,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:17,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:17,306 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:17,306 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,306 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:17,306 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:17,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,307 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,309 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,309 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,310 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,310 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:17,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:17,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:17,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:17,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:17,311 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:17,311 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,311 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:17,311 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:17,312 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:17,312 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:17,313 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,314 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,314 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,314 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:17,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c315562[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:17,316 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:17,316 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,317 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:17,317 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:17,318 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,318 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:17,321 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,321 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:17,324 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:17,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:17,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:17,324 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:17,324 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:17,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,325 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,326 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,328 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:17,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,331 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,331 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:17,331 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,331 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703837338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703837374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703837374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703837375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,432 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:17,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:17,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:17,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:17,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:17,435 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:17,435 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:17,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,436 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,437 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,437 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,438 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:17,439 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:17,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:17,440 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:17,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:17,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,440 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:17,441 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:17,442 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,442 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,442 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,442 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:17,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d429a4[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:17,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:17,444 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,445 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,446 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:17,446 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:17,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:17,446 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:17,449 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:17,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:17,451 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:17,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:17,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:17,452 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:17,452 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:17,454 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:17,454 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,454 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,455 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,456 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:17,459 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:17,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:17,459 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:17,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,460 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:17,460 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:17,460 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:17,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/7b7bbcb77eef4a328f353d8be568e7dd 2023-04-17 03:56:17,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/438b123cf60b4165b69c71a9059c5766 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766 2023-04-17 03:56:17,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766, entries=150, sequenceid=477, filesize=25.4 K 2023-04-17 03:56:17,527 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/d6c9f6d5af79498cb9768a7b39b266a7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7 2023-04-17 03:56:17,530 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7, entries=150, sequenceid=477, filesize=11.9 K 2023-04-17 03:56:17,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/7b7bbcb77eef4a328f353d8be568e7dd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd 2023-04-17 03:56:17,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd, entries=150, sequenceid=477, filesize=11.9 K 2023-04-17 03:56:17,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 41c317bf95e4bf8c59626c6f1adb8a3b in 863ms, sequenceid=477, compaction requested=true 2023-04-17 03:56:17,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:17,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:17,534 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 03:56:17,534 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 03:56:17,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:17,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:17,536 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 183004 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 03:56:17,536 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:17,536 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,536 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/970e26657660495781755bdbffbd8dbd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=178.7 K 2023-04-17 03:56:17,536 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85984 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 03:56:17,537 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:17,537 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,537 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b6b07c4710bc4c0c95f8d353f3d4ed31, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=84.0 K 2023-04-17 03:56:17,537 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 970e26657660495781755bdbffbd8dbd, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768752 2023-04-17 03:56:17,537 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting b6b07c4710bc4c0c95f8d353f3d4ed31, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768752 2023-04-17 03:56:17,537 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 4ed77c9bb77148c59250c3e4f383569e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703769374 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 91e8c60f1b774aa3ab392713d8da3137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703769374 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting aee7288306534561bec25f31a4a3a7f9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681703771540 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 17174ec04aaa4b0fa809efa505f477b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681703771540 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 2f100d9ee5ae4ffb9724f114a230ca35, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703772155 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fc55427d7000419a8fccc87cd58a4ca9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703772155 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 33c3b999e7364a59b14a91ae3215cfa6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681703773384 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c0844a8c2bb54f1a825649f1e2ceda5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681703773384 2023-04-17 03:56:17,538 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 19059b4cb73d491b8018a2c056bfed64, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703774513 2023-04-17 03:56:17,539 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1774af70db224839b22fd5345ea8a8ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703774513 2023-04-17 03:56:17,539 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 438b123cf60b4165b69c71a9059c5766, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:17,539 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting d6c9f6d5af79498cb9768a7b39b266a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:17,551 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#583 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:17,557 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#582 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:17,560 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:17,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:17,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:17,564 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:17,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:17,564 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:17,565 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,565 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,567 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:17,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:17,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,567 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:17,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:17,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:17,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:17,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:17,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:17,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:17,568 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:17,569 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:17,569 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:17,569 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-17 03:56:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:17,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:17,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703837880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703837881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703837882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703837882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:17,962 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/50641278fe6643b48e9e5c5eaca0bbb0 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/50641278fe6643b48e9e5c5eaca0bbb0 2023-04-17 03:56:17,967 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into 50641278fe6643b48e9e5c5eaca0bbb0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:17,967 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:17,967 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=9, startTime=1681703777534; duration=0sec 2023-04-17 03:56:17,967 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:17,967 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-17 03:56:17,970 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85984 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-17 03:56:17,970 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:17,970 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:17,970 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5358c89db7dc49328317a524f98b474c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=84.0 K 2023-04-17 03:56:17,970 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5358c89db7dc49328317a524f98b474c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681703768752 2023-04-17 03:56:17,971 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting c2b75fb7c79840ef9f3417ead97160ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1681703769374 2023-04-17 03:56:17,971 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 07c4284811d94130afa3286544f16dde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681703771540 2023-04-17 03:56:17,972 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4532a05f04bc4cc2bb17ee5d8e3e2939, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681703772155 2023-04-17 03:56:17,972 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 1cf0ec9bf5e746bb9f89eb4dc3d61e6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1681703773384 2023-04-17 03:56:17,973 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5b44f8b2b2a44a35a210018519d381db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681703774513 2023-04-17 03:56:17,973 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 7b7bbcb77eef4a328f353d8be568e7dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:17,975 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/79d7cbadcb084e57b408e29df58b27cc as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/79d7cbadcb084e57b408e29df58b27cc 2023-04-17 03:56:17,979 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into 79d7cbadcb084e57b408e29df58b27cc(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:17,979 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:17,979 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=9, startTime=1681703777534; duration=0sec 2023-04-17 03:56:17,980 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:17,985 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#585 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:17,988 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304174d9ddb46081642c087c73d2cffcec6bc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174d9ddb46081642c087c73d2cffcec6bc 2023-04-17 03:56:17,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:17,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703837985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,003 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=512, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/c7b96695578b4934b20088a60b34392b 2023-04-17 03:56:18,044 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/79e889dedb55490b9ccf3adba5dab200 2023-04-17 03:56:18,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703838189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,396 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/4510f7ceb3d14b999a223992307fa9ec as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4510f7ceb3d14b999a223992307fa9ec 2023-04-17 03:56:18,400 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 4510f7ceb3d14b999a223992307fa9ec(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:18,400 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:18,400 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=9, startTime=1681703777534; duration=0sec 2023-04-17 03:56:18,400 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:18,468 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/5b2d8119bf804bb395fd16ff4c89384a 2023-04-17 03:56:18,471 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/c7b96695578b4934b20088a60b34392b as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b 2023-04-17 03:56:18,477 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b, entries=150, sequenceid=512, filesize=25.4 K 2023-04-17 03:56:18,478 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/79e889dedb55490b9ccf3adba5dab200 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200 2023-04-17 03:56:18,482 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200, entries=150, sequenceid=512, filesize=11.9 K 2023-04-17 03:56:18,482 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/5b2d8119bf804bb395fd16ff4c89384a as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a 2023-04-17 03:56:18,485 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a, entries=150, sequenceid=512, filesize=11.9 K 2023-04-17 03:56:18,486 INFO [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 41c317bf95e4bf8c59626c6f1adb8a3b in 917ms, sequenceid=512, compaction requested=false 2023-04-17 03:56:18,486 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:18,486 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:18,487 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:18,490 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,490 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,491 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:18,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32854cc2[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:18,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:18,492 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,493 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:18,493 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:18,494 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:18,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,494 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:18,495 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,495 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,496 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:18,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-17 03:56:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:18,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:18,499 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:18,499 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:18,499 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:18,499 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:18,499 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:18,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,503 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,504 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,509 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,511 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417a2779cc5a3c144ca86b39883593e7bc5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a2779cc5a3c144ca86b39883593e7bc5 2023-04-17 03:56:18,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:18,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:18,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,512 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:18,512 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:18,512 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,512 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,513 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,516 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=527, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/51eb7f37abef4df58e334c21abb21ad4 2023-04-17 03:56:18,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/04f17cb41312401a8bd04e568c27996f 2023-04-17 03:56:18,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/25b954e029e740e79c1cf130afbf2b12 2023-04-17 03:56:18,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/51eb7f37abef4df58e334c21abb21ad4 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4 2023-04-17 03:56:18,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4, entries=150, sequenceid=527, filesize=25.4 K 2023-04-17 03:56:18,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/04f17cb41312401a8bd04e568c27996f as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f 2023-04-17 03:56:18,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f, entries=150, sequenceid=527, filesize=11.9 K 2023-04-17 03:56:18,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/25b954e029e740e79c1cf130afbf2b12 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12 2023-04-17 03:56:18,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12, entries=150, sequenceid=527, filesize=11.9 K 2023-04-17 03:56:18,558 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=107.34 KB/109920 for 41c317bf95e4bf8c59626c6f1adb8a3b in 61ms, sequenceid=527, compaction requested=true 2023-04-17 03:56:18,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:18,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:18,558 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:18,558 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:18,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:18,558 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:18,559 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:18,559 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:18,559 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,559 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/79d7cbadcb084e57b408e29df58b27cc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=77.2 K 2023-04-17 03:56:18,559 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:18,560 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:18,560 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,560 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/50641278fe6643b48e9e5c5eaca0bbb0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=36.6 K 2023-04-17 03:56:18,560 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 79d7cbadcb084e57b408e29df58b27cc, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:18,560 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 50641278fe6643b48e9e5c5eaca0bbb0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:18,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:18,560 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-17 03:56:18,560 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting c7b96695578b4934b20088a60b34392b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703776747 2023-04-17 03:56:18,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:18,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:18,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:18,560 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 79e889dedb55490b9ccf3adba5dab200, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703776747 2023-04-17 03:56:18,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:18,561 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 51eb7f37abef4df58e334c21abb21ad4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:18,561 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 04f17cb41312401a8bd04e568c27996f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:18,571 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#593 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:18,581 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#592 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:18,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703838598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,601 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/03ca1c64a3344a6e927be0e29602d40d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/03ca1c64a3344a6e927be0e29602d40d 2023-04-17 03:56:18,606 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into 03ca1c64a3344a6e927be0e29602d40d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:18,606 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:18,606 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=13, startTime=1681703778558; duration=0sec 2023-04-17 03:56:18,606 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:18,606 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:18,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:18,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:18,611 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,611 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4510f7ceb3d14b999a223992307fa9ec, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=36.6 K 2023-04-17 03:56:18,611 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 4510f7ceb3d14b999a223992307fa9ec, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1681703776668 2023-04-17 03:56:18,612 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 5b2d8119bf804bb395fd16ff4c89384a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1681703776747 2023-04-17 03:56:18,612 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 25b954e029e740e79c1cf130afbf2b12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:18,613 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:18,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:18,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:18,616 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:18,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:18,616 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:18,617 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,617 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,617 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#594 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:18,618 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,618 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,619 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:18,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:18,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:18,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,624 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:18,628 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,628 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:18,628 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:18,628 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:18,628 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:18,630 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,630 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:18,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9f3496c[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:18,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:18,632 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,633 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:18,633 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:18,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,633 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:18,636 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,636 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:18,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:18,638 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:18,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:18,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:18,638 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:18,638 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:18,639 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,639 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,639 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,640 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,641 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,645 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:18,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:18,645 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,646 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,646 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:18,646 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,646 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:18,646 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703838702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,746 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:18,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:18,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:18,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:18,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:18,749 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:18,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:18,749 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:18,750 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,750 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,751 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,751 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,751 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,751 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:18,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:18,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:18,753 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,753 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:18,753 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:18,753 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:18,753 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:18,755 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,755 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,756 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:18,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7f8775[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:18,757 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:18,757 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,759 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,759 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:18,759 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:18,759 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:18,764 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,765 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:18,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:18,767 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:18,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:18,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:18,767 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:18,767 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:18,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,769 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,770 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,772 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:18,774 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:18,774 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,775 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:18,775 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:18,775 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,775 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,875 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:18,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:18,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-17 03:56:18,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-17 03:56:18,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-17 03:56:18,881 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-17 03:56:18,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-17 03:56:18,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-17 03:56:18,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,881 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,883 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,884 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-17 03:56:18,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-17 03:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,885 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-17 03:56:18,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-17 03:56:18,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-17 03:56:18,885 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,885 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. started... 2023-04-17 03:56:18,885 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. as already flushing 2023-04-17 03:56:18,885 DEBUG [rs(jenkins-hbase4.apache.org,36381,1681703614253)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:18,885 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-17 03:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-17 03:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-17 03:56:18,886 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-17 03:56:18,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,887 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59880 deadline: 1681703838885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,887 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,887 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,888 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-17 03:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36381,1681703614253' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-17 03:56:18,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ee0437[Count = 0] remaining members to acquire global barrier 2023-04-17 03:56:18,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-17 03:56:18,889 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,891 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-17 03:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-17 03:56:18,891 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36381,1681703614253' in zk 2023-04-17 03:56:18,891 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,891 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-17 03:56:18,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36381] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59892 deadline: 1681703838891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,892 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,892 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-17 03:56:18,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,893 DEBUG [member: 'jenkins-hbase4.apache.org,36381,1681703614253' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-17 03:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36381,1681703614253': 2023-04-17 03:56:18,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36381,1681703614253' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-17 03:56:18,895 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-17 03:56:18,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-17 03:56:18,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-17 03:56:18,895 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-17 03:56:18,895 INFO [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-17 03:56:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36381] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59896 deadline: 1681703838894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-17 03:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-17 03:56:18,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,899 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-17 03:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,900 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,901 DEBUG [Thread-3807] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:60620 2023-04-17 03:56:18,901 DEBUG [Thread-3807] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:18,901 DEBUG [Thread-3811] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eedd898 to 127.0.0.1:60620 2023-04-17 03:56:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-17 03:56:18,901 DEBUG [Thread-3811] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:18,902 DEBUG [Thread-3813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x618c0022 to 127.0.0.1:60620 2023-04-17 03:56:18,902 DEBUG [Thread-3813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,903 DEBUG [Thread-3809] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:60620 2023-04-17 03:56:18,903 DEBUG [Thread-3809] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:18,903 DEBUG [Thread-3805] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:60620 2023-04-17 03:56:18,903 DEBUG [Thread-3805] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:18,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:18,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703838905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,907 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-17 03:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-17 03:56:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,916 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,916 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-17 03:56:18,917 DEBUG [(jenkins-hbase4.apache.org,43969,1681703612963)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-17 03:56:18,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-17 03:56:18,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-17 03:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-17 03:56:18,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-17 03:56:18,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-17 03:56:18,917 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-17 03:56:18,917 DEBUG [Thread-3804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-17 03:56:18,917 DEBUG [Thread-3804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-17 03:56:18,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-17 03:56:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-17 03:56:18,982 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304170f79995b17ee4211b810af42a01e9bb6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170f79995b17ee4211b810af42a01e9bb6 2023-04-17 03:56:19,015 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/9917045a6fe04a0eb3b8b292f38317f9 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9917045a6fe04a0eb3b8b292f38317f9 2023-04-17 03:56:19,017 DEBUG [Thread-3804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-17 03:56:19,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-17 03:56:19,019 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into 9917045a6fe04a0eb3b8b292f38317f9(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:19,019 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:19,019 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=13, startTime=1681703778558; duration=0sec 2023-04-17 03:56:19,019 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:19,033 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/8fd443e3500048508a67d1af14ffb58e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8fd443e3500048508a67d1af14ffb58e 2023-04-17 03:56:19,036 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 8fd443e3500048508a67d1af14ffb58e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:19,036 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:19,036 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=13, startTime=1681703778558; duration=0sec 2023-04-17 03:56:19,036 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:19,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:19,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703839209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:19,387 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=547, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/f8f7336247d44c46ba48c4d9744f9eea 2023-04-17 03:56:19,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:19,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59868 deadline: 1681703839693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:19,695 DEBUG [Thread-3796] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., hostname=jenkins-hbase4.apache.org,36381,1681703614253, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-17 03:56:19,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-17 03:56:19,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59916 deadline: 1681703839712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:19,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/320682981c0a4c5a84ad30d39a8b2b40 2023-04-17 03:56:20,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/fecc68ed4deb446482151b5cf3caea99 2023-04-17 03:56:20,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/f8f7336247d44c46ba48c4d9744f9eea as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea 2023-04-17 03:56:20,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea, entries=150, sequenceid=547, filesize=25.4 K 2023-04-17 03:56:20,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/320682981c0a4c5a84ad30d39a8b2b40 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40 2023-04-17 03:56:20,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40, entries=150, sequenceid=547, filesize=11.9 K 2023-04-17 03:56:20,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/fecc68ed4deb446482151b5cf3caea99 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99 2023-04-17 03:56:20,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99, entries=150, sequenceid=547, filesize=11.9 K 2023-04-17 03:56:20,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1660ms, sequenceid=547, compaction requested=false 2023-04-17 03:56:20,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36381] regionserver.HRegion(9158): Flush requested on 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:20,722 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-17 03:56:20,722 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:20,723 DEBUG [Thread-3800] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:60620 2023-04-17 03:56:20,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:20,723 DEBUG [Thread-3800] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:20,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:20,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:20,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:20,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:20,907 DEBUG [Thread-3794] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:60620 2023-04-17 03:56:20,908 DEBUG [Thread-3798] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:60620 2023-04-17 03:56:20,908 DEBUG [Thread-3794] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:20,908 DEBUG [Thread-3798] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:20,913 DEBUG [Thread-3802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:60620 2023-04-17 03:56:20,913 DEBUG [Thread-3802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:21,134 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230417431abd238bad4ac9a772e2fc8cddc4a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417431abd238bad4ac9a772e2fc8cddc4a4 2023-04-17 03:56:21,538 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=563, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/ac1ffd40a119409482e2245c16813040 2023-04-17 03:56:21,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/9c6e99186ceb4707aac662c2b0eaf118 2023-04-17 03:56:22,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=563 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/6e1613e595c44681b77355ffd07ced55 2023-04-17 03:56:22,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/ac1ffd40a119409482e2245c16813040 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040 2023-04-17 03:56:22,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040, entries=150, sequenceid=563, filesize=25.4 K 2023-04-17 03:56:22,360 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/9c6e99186ceb4707aac662c2b0eaf118 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118 2023-04-17 03:56:22,363 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118, entries=150, sequenceid=563, filesize=11.9 K 2023-04-17 03:56:22,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/6e1613e595c44681b77355ffd07ced55 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55 2023-04-17 03:56:22,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55, entries=150, sequenceid=563, filesize=11.9 K 2023-04-17 03:56:22,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=20.13 KB/20610 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1645ms, sequenceid=563, compaction requested=true 2023-04-17 03:56:22,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:22,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:22,367 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:22,368 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:22,368 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:22,368 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:22,368 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79160 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:22,368 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/A is initiating minor compaction (all files) 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/B is initiating minor compaction (all files) 2023-04-17 03:56:22,369 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/A in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:22,369 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/B in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:22,369 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9917045a6fe04a0eb3b8b292f38317f9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=77.3 K 2023-04-17 03:56:22,369 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/03ca1c64a3344a6e927be0e29602d40d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=36.7 K 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting 9917045a6fe04a0eb3b8b292f38317f9, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 03ca1c64a3344a6e927be0e29602d40d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting f8f7336247d44c46ba48c4d9744f9eea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681703778548 2023-04-17 03:56:22,369 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 320682981c0a4c5a84ad30d39a8b2b40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681703778548 2023-04-17 03:56:22,370 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] compactions.Compactor(207): Compacting ac1ffd40a119409482e2245c16813040, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681703778587 2023-04-17 03:56:22,370 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 9c6e99186ceb4707aac662c2b0eaf118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681703778587 2023-04-17 03:56:22,375 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#B#compaction#600 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:22,378 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#A#compaction#601 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:22,381 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/30ea59efb65845fe93df70243ac10f2d as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/30ea59efb65845fe93df70243ac10f2d 2023-04-17 03:56:22,384 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/B of 41c317bf95e4bf8c59626c6f1adb8a3b into 30ea59efb65845fe93df70243ac10f2d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:22,384 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:22,384 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/B, priority=13, startTime=1681703782367; duration=0sec 2023-04-17 03:56:22,385 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-17 03:56:22,385 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-17 03:56:22,388 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-17 03:56:22,388 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1912): 41c317bf95e4bf8c59626c6f1adb8a3b/C is initiating minor compaction (all files) 2023-04-17 03:56:22,388 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 41c317bf95e4bf8c59626c6f1adb8a3b/C in TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:22,388 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8fd443e3500048508a67d1af14ffb58e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55] into tmpdir=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp, totalSize=36.7 K 2023-04-17 03:56:22,388 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 8fd443e3500048508a67d1af14ffb58e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1681703777869 2023-04-17 03:56:22,388 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting fecc68ed4deb446482151b5cf3caea99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681703778548 2023-04-17 03:56:22,388 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] compactions.Compactor(207): Compacting 6e1613e595c44681b77355ffd07ced55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=563, earliestPutTs=1681703778587 2023-04-17 03:56:22,393 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] throttle.PressureAwareThroughputController(145): 41c317bf95e4bf8c59626c6f1adb8a3b#C#compaction#602 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-17 03:56:22,399 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/06e1dd875fab479383d406837e20b2d7 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/06e1dd875fab479383d406837e20b2d7 2023-04-17 03:56:22,402 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/C of 41c317bf95e4bf8c59626c6f1adb8a3b into 06e1dd875fab479383d406837e20b2d7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:22,402 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:22,402 INFO [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/C, priority=13, startTime=1681703782368; duration=0sec 2023-04-17 03:56:22,402 DEBUG [RS:0;jenkins-hbase4:36381-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:23,191 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/0a9792270e1c4a4996a16894837fef01 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0a9792270e1c4a4996a16894837fef01 2023-04-17 03:56:23,194 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 41c317bf95e4bf8c59626c6f1adb8a3b/A of 41c317bf95e4bf8c59626c6f1adb8a3b into 0a9792270e1c4a4996a16894837fef01(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-17 03:56:23,195 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:23,195 INFO [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b., storeName=41c317bf95e4bf8c59626c6f1adb8a3b/A, priority=13, startTime=1681703782367; duration=0sec 2023-04-17 03:56:23,195 DEBUG [RS:0;jenkins-hbase4:36381-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-17 03:56:29,791 DEBUG [Thread-3796] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:60620 2023-04-17 03:56:29,791 DEBUG [Thread-3796] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:29,791 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-17 03:56:29,791 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-17 03:56:29,791 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4173 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4116 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4124 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4155 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(398): read 4087 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-17 03:56:29,792 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:56:29,792 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:60620 2023-04-17 03:56:29,792 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:29,793 INFO [Listener at localhost/44831] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-17 03:56:29,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-17 03:56:29,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:29,796 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703789796"}]},"ts":"1681703789796"} 2023-04-17 03:56:29,796 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-17 03:56:29,798 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-17 03:56:29,798 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, UNASSIGN}] 2023-04-17 03:56:29,799 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, UNASSIGN 2023-04-17 03:56:29,800 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:29,800 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703789800"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681703789800"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681703789800"}]},"ts":"1681703789800"} 2023-04-17 03:56:29,801 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253}] 2023-04-17 03:56:29,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:29,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 41c317bf95e4bf8c59626c6f1adb8a3b, disabling compactions & flushes 2023-04-17 03:56:29,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. after waiting 0 ms 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:29,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 41c317bf95e4bf8c59626c6f1adb8a3b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=A 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:29,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=B 2023-04-17 03:56:29,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:29,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 41c317bf95e4bf8c59626c6f1adb8a3b, store=C 2023-04-17 03:56:29,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-17 03:56:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:30,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304172e0b76a7960d431a965ebd5d4a669c16 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172e0b76a7960d431a965ebd5d4a669c16 2023-04-17 03:56:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:30,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=573, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a10751398c084cc587394c6262a8e6bf 2023-04-17 03:56:30,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:31,175 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/b0700918ee7442ec9a69be349f3b014e 2023-04-17 03:56:31,583 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b8c4ab8dea654dd685268ccc4b6956cd 2023-04-17 03:56:31,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/A/a10751398c084cc587394c6262a8e6bf as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a10751398c084cc587394c6262a8e6bf 2023-04-17 03:56:31,589 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a10751398c084cc587394c6262a8e6bf, entries=100, sequenceid=573, filesize=18.6 K 2023-04-17 03:56:31,589 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/B/b0700918ee7442ec9a69be349f3b014e as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b0700918ee7442ec9a69be349f3b014e 2023-04-17 03:56:31,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b0700918ee7442ec9a69be349f3b014e, entries=100, sequenceid=573, filesize=9.5 K 2023-04-17 03:56:31,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/.tmp/C/b8c4ab8dea654dd685268ccc4b6956cd as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b8c4ab8dea654dd685268ccc4b6956cd 2023-04-17 03:56:31,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b8c4ab8dea654dd685268ccc4b6956cd, entries=100, sequenceid=573, filesize=9.5 K 2023-04-17 03:56:31,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 41c317bf95e4bf8c59626c6f1adb8a3b in 1644ms, sequenceid=573, compaction requested=false 2023-04-17 03:56:31,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/226d11695e8d4df5b1a2011b4a7fba7b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac033cfd048e4bfda444acdd8de821dc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c2b2cfd515b24ff2a0b78356550fc9bc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/970e26657660495781755bdbffbd8dbd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/79d7cbadcb084e57b408e29df58b27cc, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9917045a6fe04a0eb3b8b292f38317f9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040] to archive 2023-04-17 03:56:31,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:56:31,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/62d4451c4e064cb28354bd4f93607181 2023-04-17 03:56:31,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c262573fd6d44153bdc1ff2bd8d49625 2023-04-17 03:56:31,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/226d11695e8d4df5b1a2011b4a7fba7b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/226d11695e8d4df5b1a2011b4a7fba7b 2023-04-17 03:56:31,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/583081cea72c48eb8ac96da916e072aa 2023-04-17 03:56:31,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/8092e1f44a604e948d2f9317cbc33a9f 2023-04-17 03:56:31,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9b7f5cdc374747e9bac595c84ccc0dc6 2023-04-17 03:56:31,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/96f7d80ed15f45e68884a77f8fac65df 2023-04-17 03:56:31,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/b24f58f84cbc42b6bacc2eed74e0362b 2023-04-17 03:56:31,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac033cfd048e4bfda444acdd8de821dc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac033cfd048e4bfda444acdd8de821dc 2023-04-17 03:56:31,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a0c3358b095d4fd193060adbec78c80d 2023-04-17 03:56:31,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c2b2cfd515b24ff2a0b78356550fc9bc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c2b2cfd515b24ff2a0b78356550fc9bc 2023-04-17 03:56:31,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/6e3fe7432b954c55bda8bd40ffa562af 2023-04-17 03:56:31,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f78f755cb7ef4cbe9a760697130afc0c 2023-04-17 03:56:31,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/286c626ca085473aa1c800363ca16e61 2023-04-17 03:56:31,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/462dd707593c4510a2e87ca3203f4ac0 2023-04-17 03:56:31,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0d10a7c8bef140b89831273c8ef3f2a9 2023-04-17 03:56:31,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a4fb9392f3104d2aab7beab8020b719d 2023-04-17 03:56:31,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/36cedddd05154932b36d251d3612eac9 2023-04-17 03:56:31,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/47fa6fc411a24689b6dbc46a4f6cd1e2 2023-04-17 03:56:31,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/cbbac9c10dcf48e188657e8a88d161e8 2023-04-17 03:56:31,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/970e26657660495781755bdbffbd8dbd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/970e26657660495781755bdbffbd8dbd 2023-04-17 03:56:31,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/4ed77c9bb77148c59250c3e4f383569e 2023-04-17 03:56:31,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/aee7288306534561bec25f31a4a3a7f9 2023-04-17 03:56:31,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/2f100d9ee5ae4ffb9724f114a230ca35 2023-04-17 03:56:31,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/33c3b999e7364a59b14a91ae3215cfa6 2023-04-17 03:56:31,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/19059b4cb73d491b8018a2c056bfed64 2023-04-17 03:56:31,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/79d7cbadcb084e57b408e29df58b27cc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/79d7cbadcb084e57b408e29df58b27cc 2023-04-17 03:56:31,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/438b123cf60b4165b69c71a9059c5766 2023-04-17 03:56:31,645 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/c7b96695578b4934b20088a60b34392b 2023-04-17 03:56:31,646 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9917045a6fe04a0eb3b8b292f38317f9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/9917045a6fe04a0eb3b8b292f38317f9 2023-04-17 03:56:31,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/51eb7f37abef4df58e334c21abb21ad4 2023-04-17 03:56:31,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/f8f7336247d44c46ba48c4d9744f9eea 2023-04-17 03:56:31,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/ac1ffd40a119409482e2245c16813040 2023-04-17 03:56:31,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/081bd0adc84a4e7aa6e0d4d0f03952fd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/da0ae208b6e54359928386cd1b3a3284, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/deee300b34644fe8aa482523b68b9618, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b6b07c4710bc4c0c95f8d353f3d4ed31, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/50641278fe6643b48e9e5c5eaca0bbb0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/03ca1c64a3344a6e927be0e29602d40d, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118] to archive 2023-04-17 03:56:31,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:56:31,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/4040b428968549349ea78801c213b593 2023-04-17 03:56:31,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c3346b9c4c4a149fc6a780c27bb440 2023-04-17 03:56:31,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/081bd0adc84a4e7aa6e0d4d0f03952fd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/081bd0adc84a4e7aa6e0d4d0f03952fd 2023-04-17 03:56:31,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/52fdb5c8f28346d8b26b708459908b6f 2023-04-17 03:56:31,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2f421d9ae1d94f04976be799e17feef2 2023-04-17 03:56:31,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/2d2aa1339b744b57a1525f607fad8dcd 2023-04-17 03:56:31,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/41d56d1295864c9cb7b3d472a98b89d2 2023-04-17 03:56:31,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/da0ae208b6e54359928386cd1b3a3284 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/da0ae208b6e54359928386cd1b3a3284 2023-04-17 03:56:31,659 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/ef6216ef5b7d4927a594320150464141 2023-04-17 03:56:31,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/3c072ab0b9124cdf8a4ec409d056cc94 2023-04-17 03:56:31,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/deee300b34644fe8aa482523b68b9618 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/deee300b34644fe8aa482523b68b9618 2023-04-17 03:56:31,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/7e535735b793403f8d6517ddd1e43cfd 2023-04-17 03:56:31,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/97c1455569c24b42bc127dd3dcf5317a 2023-04-17 03:56:31,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/047d1db55a0a4985b072903482dec526 2023-04-17 03:56:31,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/dfd332dd069643a5a8fbb20a56ced617 2023-04-17 03:56:31,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/8e8837f677d74ef89433337fd05876a2 2023-04-17 03:56:31,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/681f2c38f3724f668ae77f76d1662765 2023-04-17 03:56:31,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/53353788463c4c29b97825a5413855ad 2023-04-17 03:56:31,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c8b29de07b4141418aabebe62d716c8e 2023-04-17 03:56:31,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b6b07c4710bc4c0c95f8d353f3d4ed31 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b6b07c4710bc4c0c95f8d353f3d4ed31 2023-04-17 03:56:31,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/48695568ecfa4d4c8332d5ac9701502f 2023-04-17 03:56:31,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/91e8c60f1b774aa3ab392713d8da3137 2023-04-17 03:56:31,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/17174ec04aaa4b0fa809efa505f477b8 2023-04-17 03:56:31,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/fc55427d7000419a8fccc87cd58a4ca9 2023-04-17 03:56:31,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/c0844a8c2bb54f1a825649f1e2ceda5e 2023-04-17 03:56:31,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/1774af70db224839b22fd5345ea8a8ff 2023-04-17 03:56:31,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/50641278fe6643b48e9e5c5eaca0bbb0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/50641278fe6643b48e9e5c5eaca0bbb0 2023-04-17 03:56:31,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/d6c9f6d5af79498cb9768a7b39b266a7 2023-04-17 03:56:31,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/79e889dedb55490b9ccf3adba5dab200 2023-04-17 03:56:31,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/03ca1c64a3344a6e927be0e29602d40d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/03ca1c64a3344a6e927be0e29602d40d 2023-04-17 03:56:31,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/04f17cb41312401a8bd04e568c27996f 2023-04-17 03:56:31,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/320682981c0a4c5a84ad30d39a8b2b40 2023-04-17 03:56:31,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/9c6e99186ceb4707aac662c2b0eaf118 2023-04-17 03:56:31,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/ce08ade35d474a82a83ca5afc287f717, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/0894866ac00a4d0abd2d10b69a6646a4, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/53bcb6b2b0b347ca992e4ae785519ae2, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5358c89db7dc49328317a524f98b474c, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4510f7ceb3d14b999a223992307fa9ec, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8fd443e3500048508a67d1af14ffb58e, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55] to archive 2023-04-17 03:56:31,681 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-17 03:56:31,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/e67bf34fcec44852897731942849d269 2023-04-17 03:56:31,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/a95102f303794c9381771b2c1a447aef 2023-04-17 03:56:31,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/ce08ade35d474a82a83ca5afc287f717 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/ce08ade35d474a82a83ca5afc287f717 2023-04-17 03:56:31,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fc7511c18e07465fa0b8d76188056253 2023-04-17 03:56:31,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/eeebe55a09fc482b9e4a356a1c4fb42f 2023-04-17 03:56:31,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/db83c4f67c464bef986f25f45b3fd7f0 2023-04-17 03:56:31,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b5c22bdfd6724317b18a01c01ab6a01f 2023-04-17 03:56:31,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/0894866ac00a4d0abd2d10b69a6646a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/0894866ac00a4d0abd2d10b69a6646a4 2023-04-17 03:56:31,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8e073b8c617645eb84b978eff33de1c2 2023-04-17 03:56:31,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/76de685ca446497f901b1d2a827083f7 2023-04-17 03:56:31,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/53bcb6b2b0b347ca992e4ae785519ae2 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/53bcb6b2b0b347ca992e4ae785519ae2 2023-04-17 03:56:31,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/03303405fbcc41968cb90171b43d19fd 2023-04-17 03:56:31,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/73adb9fd77f4433b96ebb9210e672949 2023-04-17 03:56:31,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b28d13a5cf034ce2864642202052cd18 2023-04-17 03:56:31,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/593ebe12a5cc441abf41ac61a5748b6e 2023-04-17 03:56:31,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/d19f1b7ee6164defbd34d49944e5f326 2023-04-17 03:56:31,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8dd5394f3ade4a01877d43c10edd2735 2023-04-17 03:56:31,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1ee5188f6b6d41bfa35f3f13b07a5016 2023-04-17 03:56:31,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4d7055279dd14ee49594140f8b96792a 2023-04-17 03:56:31,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5358c89db7dc49328317a524f98b474c to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5358c89db7dc49328317a524f98b474c 2023-04-17 03:56:31,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1b31b705d4574eacb596ecef031b8167 2023-04-17 03:56:31,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/c2b75fb7c79840ef9f3417ead97160ff 2023-04-17 03:56:31,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/07c4284811d94130afa3286544f16dde 2023-04-17 03:56:31,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4532a05f04bc4cc2bb17ee5d8e3e2939 2023-04-17 03:56:31,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/1cf0ec9bf5e746bb9f89eb4dc3d61e6e 2023-04-17 03:56:31,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b44f8b2b2a44a35a210018519d381db 2023-04-17 03:56:31,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4510f7ceb3d14b999a223992307fa9ec to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/4510f7ceb3d14b999a223992307fa9ec 2023-04-17 03:56:31,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/7b7bbcb77eef4a328f353d8be568e7dd 2023-04-17 03:56:31,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/5b2d8119bf804bb395fd16ff4c89384a 2023-04-17 03:56:31,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8fd443e3500048508a67d1af14ffb58e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/8fd443e3500048508a67d1af14ffb58e 2023-04-17 03:56:31,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/25b954e029e740e79c1cf130afbf2b12 2023-04-17 03:56:31,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/fecc68ed4deb446482151b5cf3caea99 2023-04-17 03:56:31,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/6e1613e595c44681b77355ffd07ced55 2023-04-17 03:56:31,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits/576.seqid, newMaxSeqId=576, maxSeqId=4 2023-04-17 03:56:31,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b. 2023-04-17 03:56:31,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 41c317bf95e4bf8c59626c6f1adb8a3b: 2023-04-17 03:56:31,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:31,715 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=41c317bf95e4bf8c59626c6f1adb8a3b, regionState=CLOSED 2023-04-17 03:56:31,715 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681703791715"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681703791715"}]},"ts":"1681703791715"} 2023-04-17 03:56:31,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-17 03:56:31,716 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 41c317bf95e4bf8c59626c6f1adb8a3b, server=jenkins-hbase4.apache.org,36381,1681703614253 in 1.9150 sec 2023-04-17 03:56:31,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-17 03:56:31,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=41c317bf95e4bf8c59626c6f1adb8a3b, UNASSIGN in 1.9180 sec 2023-04-17 03:56:31,718 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681703791718"}]},"ts":"1681703791718"} 2023-04-17 03:56:31,718 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-17 03:56:31,720 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-17 03:56:31,722 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9280 sec 2023-04-17 03:56:31,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-17 03:56:31,900 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-17 03:56:31,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-17 03:56:31,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,903 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 03:56:31,903 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,908 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:31,910 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C, FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits] 2023-04-17 03:56:31,912 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0a9792270e1c4a4996a16894837fef01 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/0a9792270e1c4a4996a16894837fef01 2023-04-17 03:56:31,913 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a10751398c084cc587394c6262a8e6bf to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/A/a10751398c084cc587394c6262a8e6bf 2023-04-17 03:56:31,915 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/30ea59efb65845fe93df70243ac10f2d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/30ea59efb65845fe93df70243ac10f2d 2023-04-17 03:56:31,916 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b0700918ee7442ec9a69be349f3b014e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/B/b0700918ee7442ec9a69be349f3b014e 2023-04-17 03:56:31,918 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/06e1dd875fab479383d406837e20b2d7 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/06e1dd875fab479383d406837e20b2d7 2023-04-17 03:56:31,919 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b8c4ab8dea654dd685268ccc4b6956cd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/C/b8c4ab8dea654dd685268ccc4b6956cd 2023-04-17 03:56:31,921 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits/576.seqid to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b/recovered.edits/576.seqid 2023-04-17 03:56:31,921 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/.tmp/data/default/TestAcidGuarantees/41c317bf95e4bf8c59626c6f1adb8a3b 2023-04-17 03:56:31,921 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-17 03:56:31,922 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:56:31,922 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-17 03:56:31,925 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170951d300914a4b0bbb9ad58edfddb429 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170951d300914a4b0bbb9ad58edfddb429 2023-04-17 03:56:31,925 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170f79995b17ee4211b810af42a01e9bb6 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304170f79995b17ee4211b810af42a01e9bb6 2023-04-17 03:56:31,926 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417154548a4d4b2430ea656dad53087095e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417154548a4d4b2430ea656dad53087095e 2023-04-17 03:56:31,927 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171902e9f9b24d4a0a85fa73127f0bb5f3 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171902e9f9b24d4a0a85fa73127f0bb5f3 2023-04-17 03:56:31,928 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171b4f217cc3a849d7b34295e3c1d47798 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304171b4f217cc3a849d7b34295e3c1d47798 2023-04-17 03:56:31,929 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172e0b76a7960d431a965ebd5d4a669c16 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304172e0b76a7960d431a965ebd5d4a669c16 2023-04-17 03:56:31,930 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041730881ed77e6a4969bca215aef770b132 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041730881ed77e6a4969bca215aef770b132 2023-04-17 03:56:31,931 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041738ad1afd84f64732b4ddc2768f1ee88d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041738ad1afd84f64732b4ddc2768f1ee88d 2023-04-17 03:56:31,931 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173a44df3fc9ef4e4f96cc4fe5e9dad637 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304173a44df3fc9ef4e4f96cc4fe5e9dad637 2023-04-17 03:56:31,932 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740083152266e4e17b6f94ea88a4adcbd to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041740083152266e4e17b6f94ea88a4adcbd 2023-04-17 03:56:31,933 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417431abd238bad4ac9a772e2fc8cddc4a4 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417431abd238bad4ac9a772e2fc8cddc4a4 2023-04-17 03:56:31,934 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174d9ddb46081642c087c73d2cffcec6bc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304174d9ddb46081642c087c73d2cffcec6bc 2023-04-17 03:56:31,935 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175ff2139e97e14649bf62d895508f617e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304175ff2139e97e14649bf62d895508f617e 2023-04-17 03:56:31,935 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041766517ccbe3fe4c54b78820d82dd3b9fb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041766517ccbe3fe4c54b78820d82dd3b9fb 2023-04-17 03:56:31,936 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176c0e95d9a37c47ad8ca38d599387f4cb to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304176c0e95d9a37c47ad8ca38d599387f4cb 2023-04-17 03:56:31,937 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177493d6eab34141d1b3b6587a5e9de1e1 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177493d6eab34141d1b3b6587a5e9de1e1 2023-04-17 03:56:31,938 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b385b9f35a7482cb7e2781ab36e9d5d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b385b9f35a7482cb7e2781ab36e9d5d 2023-04-17 03:56:31,939 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b5e3ed3bc094b758a5c3d71b6848efc to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304177b5e3ed3bc094b758a5c3d71b6848efc 2023-04-17 03:56:31,940 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417811d89df25c04a5c939369ec50ceae56 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417811d89df25c04a5c939369ec50ceae56 2023-04-17 03:56:31,941 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041794e5f79636394e0792441d88f344949e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041794e5f79636394e0792441d88f344949e 2023-04-17 03:56:31,941 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417982d20db856045ac8709f5b840de85e5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417982d20db856045ac8709f5b840de85e5 2023-04-17 03:56:31,942 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179ba1d6cb45804d389100476d80fef360 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179ba1d6cb45804d389100476d80fef360 2023-04-17 03:56:31,943 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179e7cdcbe98d241d69df32daf23e2c51d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304179e7cdcbe98d241d69df32daf23e2c51d 2023-04-17 03:56:31,944 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a2779cc5a3c144ca86b39883593e7bc5 to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a2779cc5a3c144ca86b39883593e7bc5 2023-04-17 03:56:31,944 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a6ab9a9fd71a4cd2870ed12b820e341d to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417a6ab9a9fd71a4cd2870ed12b820e341d 2023-04-17 03:56:31,945 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ded7d173d4624823ae027c07b85f185b to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417ded7d173d4624823ae027c07b85f185b 2023-04-17 03:56:31,946 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e94b8c45e7454134b1d2935cc326fd3e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417e94b8c45e7454134b1d2935cc326fd3e 2023-04-17 03:56:31,947 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417fb03f8b23a30458fa0861589f03a323e to hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230417fb03f8b23a30458fa0861589f03a323e 2023-04-17 03:56:31,947 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-17 03:56:31,949 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,950 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-17 03:56:31,952 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-17 03:56:31,953 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,953 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-17 03:56:31,953 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681703791953"}]},"ts":"9223372036854775807"} 2023-04-17 03:56:31,954 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-17 03:56:31,954 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 41c317bf95e4bf8c59626c6f1adb8a3b, NAME => 'TestAcidGuarantees,,1681703757314.41c317bf95e4bf8c59626c6f1adb8a3b.', STARTKEY => '', ENDKEY => ''}] 2023-04-17 03:56:31,954 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-17 03:56:31,954 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681703791954"}]},"ts":"9223372036854775807"} 2023-04-17 03:56:31,956 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-17 03:56:31,957 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-17 03:56:31,958 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-04-17 03:56:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43969] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-17 03:56:32,004 INFO [Listener at localhost/44831] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-17 03:56:32,015 INFO [Listener at localhost/44831] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 235), OpenFileDescriptor=551 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=373 (was 391), ProcessCount=169 (was 169), AvailableMemoryMB=1092 (was 986) - AvailableMemoryMB LEAK? - 2023-04-17 03:56:32,015 INFO [Listener at localhost/44831] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-17 03:56:32,016 INFO [Listener at localhost/44831] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-17 03:56:32,016 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:60620 2023-04-17 03:56:32,016 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:32,017 DEBUG [Listener at localhost/44831] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-17 03:56:32,017 DEBUG [Listener at localhost/44831] util.JVMClusterUtil(257): Found active master hash=457972121, stopped=false 2023-04-17 03:56:32,017 INFO [Listener at localhost/44831] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:56:32,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 03:56:32,018 INFO [Listener at localhost/44831] procedure2.ProcedureExecutor(629): Stopping 2023-04-17 03:56:32,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-17 03:56:32,018 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:56:32,018 DEBUG [Listener at localhost/44831] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1266022f to 127.0.0.1:60620 2023-04-17 03:56:32,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 03:56:32,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-17 03:56:32,018 DEBUG [Listener at localhost/44831] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:32,018 INFO [Listener at localhost/44831] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36381,1681703614253' ***** 2023-04-17 03:56:32,019 INFO [Listener at localhost/44831] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-17 03:56:32,019 INFO [RS:0;jenkins-hbase4:36381] regionserver.HeapMemoryManager(220): Stopping 2023-04-17 03:56:32,019 INFO [RS:0;jenkins-hbase4:36381] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-17 03:56:32,019 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-17 03:56:32,019 INFO [RS:0;jenkins-hbase4:36381] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-17 03:56:32,019 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(3304): Received CLOSE for af5b89de65a7835500f53c143436eb3a 2023-04-17 03:56:32,020 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:32,020 DEBUG [RS:0;jenkins-hbase4:36381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b0dd69d to 127.0.0.1:60620 2023-04-17 03:56:32,020 DEBUG [RS:0;jenkins-hbase4:36381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:32,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing af5b89de65a7835500f53c143436eb3a, disabling compactions & flushes 2023-04-17 03:56:32,021 INFO [RS:0;jenkins-hbase4:36381] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-17 03:56:32,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:56:32,021 INFO [RS:0;jenkins-hbase4:36381] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-17 03:56:32,021 INFO [RS:0;jenkins-hbase4:36381] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-17 03:56:32,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:56:32,021 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-17 03:56:32,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. after waiting 0 ms 2023-04-17 03:56:32,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:56:32,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing af5b89de65a7835500f53c143436eb3a 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-17 03:56:32,021 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-17 03:56:32,021 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1479): Online Regions={af5b89de65a7835500f53c143436eb3a=hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a., 1588230740=hbase:meta,,1.1588230740} 2023-04-17 03:56:32,022 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-17 03:56:32,022 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-17 03:56:32,022 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-17 03:56:32,022 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-17 03:56:32,022 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-17 03:56:32,022 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-17 03:56:32,023 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1505): Waiting on 1588230740, af5b89de65a7835500f53c143436eb3a 2023-04-17 03:56:32,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/.tmp/info/25d8a3aa71ca4321bb134cd9126ecf57 2023-04-17 03:56:32,029 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/info/2d4e25bd4ccb4245be909551bf3bc960 2023-04-17 03:56:32,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/.tmp/info/25d8a3aa71ca4321bb134cd9126ecf57 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/info/25d8a3aa71ca4321bb134cd9126ecf57 2023-04-17 03:56:32,033 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2d4e25bd4ccb4245be909551bf3bc960 2023-04-17 03:56:32,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/info/25d8a3aa71ca4321bb134cd9126ecf57, entries=2, sequenceid=6, filesize=4.8 K 2023-04-17 03:56:32,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for af5b89de65a7835500f53c143436eb3a in 16ms, sequenceid=6, compaction requested=false 2023-04-17 03:56:32,038 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-17 03:56:32,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/rep_barrier/53e399abd71741259a3db3f4d7a1f31c 2023-04-17 03:56:32,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/namespace/af5b89de65a7835500f53c143436eb3a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-17 03:56:32,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:56:32,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for af5b89de65a7835500f53c143436eb3a: 2023-04-17 03:56:32,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681703616651.af5b89de65a7835500f53c143436eb3a. 2023-04-17 03:56:32,049 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 53e399abd71741259a3db3f4d7a1f31c 2023-04-17 03:56:32,091 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-17 03:56:32,091 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-17 03:56:32,224 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 03:56:32,424 DEBUG [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-17 03:56:32,454 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/table/be6e4b608f244082b85e6cf065842309 2023-04-17 03:56:32,457 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for be6e4b608f244082b85e6cf065842309 2023-04-17 03:56:32,458 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/info/2d4e25bd4ccb4245be909551bf3bc960 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/info/2d4e25bd4ccb4245be909551bf3bc960 2023-04-17 03:56:32,461 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2d4e25bd4ccb4245be909551bf3bc960 2023-04-17 03:56:32,461 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/info/2d4e25bd4ccb4245be909551bf3bc960, entries=22, sequenceid=93, filesize=7.1 K 2023-04-17 03:56:32,462 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/rep_barrier/53e399abd71741259a3db3f4d7a1f31c as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/rep_barrier/53e399abd71741259a3db3f4d7a1f31c 2023-04-17 03:56:32,465 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 53e399abd71741259a3db3f4d7a1f31c 2023-04-17 03:56:32,465 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/rep_barrier/53e399abd71741259a3db3f4d7a1f31c, entries=6, sequenceid=93, filesize=5.5 K 2023-04-17 03:56:32,466 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/.tmp/table/be6e4b608f244082b85e6cf065842309 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/table/be6e4b608f244082b85e6cf065842309 2023-04-17 03:56:32,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for be6e4b608f244082b85e6cf065842309 2023-04-17 03:56:32,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/table/be6e4b608f244082b85e6cf065842309, entries=9, sequenceid=93, filesize=5.5 K 2023-04-17 03:56:32,469 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 447ms, sequenceid=93, compaction requested=false 2023-04-17 03:56:32,476 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-17 03:56:32,477 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 03:56:32,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-17 03:56:32,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-17 03:56:32,478 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-17 03:56:32,624 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36381,1681703614253; all regions closed. 2023-04-17 03:56:32,632 DEBUG [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/oldWALs 2023-04-17 03:56:32,632 INFO [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36381%2C1681703614253.meta:.meta(num 1681703616380) 2023-04-17 03:56:32,636 DEBUG [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/oldWALs 2023-04-17 03:56:32,636 INFO [RS:0;jenkins-hbase4:36381] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36381%2C1681703614253:(num 1681703616248) 2023-04-17 03:56:32,636 DEBUG [RS:0;jenkins-hbase4:36381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:32,636 INFO [RS:0;jenkins-hbase4:36381] regionserver.LeaseManager(133): Closed leases 2023-04-17 03:56:32,636 INFO [RS:0;jenkins-hbase4:36381] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-17 03:56:32,636 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 03:56:32,638 INFO [RS:0;jenkins-hbase4:36381] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36381 2023-04-17 03:56:32,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36381,1681703614253 2023-04-17 03:56:32,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 03:56:32,643 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-17 03:56:32,643 ERROR [Listener at localhost/44831-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@16af48b1 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2422] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 03:56:32,645 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36381,1681703614253] 2023-04-17 03:56:32,646 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36381,1681703614253; numProcessing=1 2023-04-17 03:56:32,647 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36381,1681703614253 already deleted, retry=false 2023-04-17 03:56:32,647 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36381,1681703614253 expired; onlineServers=0 2023-04-17 03:56:32,647 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43969,1681703612963' ***** 2023-04-17 03:56:32,647 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-17 03:56:32,647 DEBUG [M:0;jenkins-hbase4:43969] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a7113a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-17 03:56:32,647 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43969,1681703612963 2023-04-17 03:56:32,647 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43969,1681703612963; all regions closed. 2023-04-17 03:56:32,647 DEBUG [M:0;jenkins-hbase4:43969] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-17 03:56:32,647 DEBUG [M:0;jenkins-hbase4:43969] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-17 03:56:32,648 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-17 03:56:32,648 DEBUG [M:0;jenkins-hbase4:43969] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-17 03:56:32,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681703615850] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681703615850,5,FailOnTimeoutGroup] 2023-04-17 03:56:32,648 INFO [M:0;jenkins-hbase4:43969] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-17 03:56:32,648 INFO [M:0;jenkins-hbase4:43969] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-17 03:56:32,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681703615861] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681703615861,5,FailOnTimeoutGroup] 2023-04-17 03:56:32,648 INFO [M:0;jenkins-hbase4:43969] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-17 03:56:32,648 DEBUG [M:0;jenkins-hbase4:43969] master.HMaster(1512): Stopping service threads 2023-04-17 03:56:32,648 INFO [M:0;jenkins-hbase4:43969] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-17 03:56:32,649 ERROR [M:0;jenkins-hbase4:43969] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (54741785) connection to localhost/127.0.0.1:40881 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-17 03:56:32,649 INFO [M:0;jenkins-hbase4:43969] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-17 03:56:32,649 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-17 03:56:32,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-17 03:56:32,651 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-17 03:56:32,651 DEBUG [M:0;jenkins-hbase4:43969] zookeeper.ZKUtil(398): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-17 03:56:32,651 WARN [M:0;jenkins-hbase4:43969] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-17 03:56:32,651 INFO [M:0;jenkins-hbase4:43969] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-17 03:56:32,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-17 03:56:32,651 INFO [M:0;jenkins-hbase4:43969] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-17 03:56:32,652 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-17 03:56:32,652 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:56:32,652 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:56:32,652 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-17 03:56:32,652 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:56:32,652 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.01 KB heapSize=393.68 KB 2023-04-17 03:56:32,665 INFO [M:0;jenkins-hbase4:43969] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.01 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d024642c83004870bf774858fb51d446 2023-04-17 03:56:32,670 INFO [M:0;jenkins-hbase4:43969] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d024642c83004870bf774858fb51d446 2023-04-17 03:56:32,670 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d024642c83004870bf774858fb51d446 as hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d024642c83004870bf774858fb51d446 2023-04-17 03:56:32,673 INFO [M:0;jenkins-hbase4:43969] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d024642c83004870bf774858fb51d446 2023-04-17 03:56:32,673 INFO [M:0;jenkins-hbase4:43969] regionserver.HStore(1080): Added hdfs://localhost:40881/user/jenkins/test-data/43e1b9d5-9c84-d431-722e-ae450a8a3ad9/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d024642c83004870bf774858fb51d446, entries=65, sequenceid=578, filesize=24.7 K 2023-04-17 03:56:32,674 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegion(2948): Finished flush of dataSize ~345.01 KB/353295, heapSize ~393.66 KB/403112, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 22ms, sequenceid=578, compaction requested=false 2023-04-17 03:56:32,676 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-17 03:56:32,676 DEBUG [M:0;jenkins-hbase4:43969] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-17 03:56:32,679 INFO [M:0;jenkins-hbase4:43969] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-17 03:56:32,679 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-17 03:56:32,680 INFO [M:0;jenkins-hbase4:43969] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43969 2023-04-17 03:56:32,681 DEBUG [M:0;jenkins-hbase4:43969] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43969,1681703612963 already deleted, retry=false 2023-04-17 03:56:32,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 03:56:32,747 INFO [RS:0;jenkins-hbase4:36381] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36381,1681703614253; zookeeper connection closed. 2023-04-17 03:56:32,747 ERROR [Listener at localhost/44831-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a094b61 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2422] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 03:56:32,747 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): regionserver:36381-0x101904098950001, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 03:56:32,747 ERROR [Listener at localhost/44831-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@727e2712 rejected from java.util.concurrent.ThreadPoolExecutor@1290debc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2422] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 03:56:32,747 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@b95cba4] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@b95cba4 2023-04-17 03:56:32,747 INFO [Listener at localhost/44831] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-17 03:56:32,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 03:56:32,847 INFO [M:0;jenkins-hbase4:43969] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43969,1681703612963; zookeeper connection closed. 2023-04-17 03:56:32,847 ERROR [Listener at localhost/44831-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3046a9b2 rejected from java.util.concurrent.ThreadPoolExecutor@e1350a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4038] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 03:56:32,847 DEBUG [Listener at localhost/44831-EventThread] zookeeper.ZKWatcher(600): master:43969-0x101904098950000, quorum=127.0.0.1:60620, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-17 03:56:32,847 ERROR [Listener at localhost/44831-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41b4079 rejected from java.util.concurrent.ThreadPoolExecutor@e1350a7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4038] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-17 03:56:32,849 WARN [Listener at localhost/44831] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-17 03:56:32,852 INFO [Listener at localhost/44831] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-17 03:56:32,961 WARN [BP-1537323972-172.31.14.131-1681703609603 heartbeating to localhost/127.0.0.1:40881] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-17 03:56:32,961 WARN [BP-1537323972-172.31.14.131-1681703609603 heartbeating to localhost/127.0.0.1:40881] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1537323972-172.31.14.131-1681703609603 (Datanode Uuid 4d8c19e9-6c86-45aa-a882-f64680513502) service to localhost/127.0.0.1:40881 2023-04-17 03:56:32,964 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787/dfs/data/data1/current/BP-1537323972-172.31.14.131-1681703609603] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 03:56:32,965 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/04d1e375-44d9-fc04-431e-3796bd2a97ba/cluster_6b2c4ab0-e173-184c-6d78-2cddef0bd787/dfs/data/data2/current/BP-1537323972-172.31.14.131-1681703609603] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-17 03:56:33,000 INFO [Listener at localhost/44831] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-17 03:56:33,132 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-17 03:56:33,170 INFO [Listener at localhost/44831] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-17 03:56:33,349 INFO [Listener at localhost/44831] hbase.HBaseTestingUtility(1293): Minicluster is down